README.md

---
license: apache-2.0
base_model: openai/gpt-oss-120b
library_name: mlx
pipeline_tag: text-generation
tags:
- mlx
- omlx
- gpt_oss
- quantized
- oQ4
- apple-silicon
- 4-bit
---

# gpt-oss-120b-oQ4

`cjnielson44/gpt-oss-120b-oQ4` is an Apple Silicon / oMLX-ready MLX checkpoint for GPT-OSS 120B. It was produced with oMLX oQ4 quantization and published for local inference through oMLX.

This checkpoint is not a uniform 4-bit conversion of every tensor. GPT-OSS uses MoE expert projection tensors that are already stored in MXFP4 form, so those expert tensors are preserved as MXFP4 passthrough tensors and explicitly marked in `config.json`.

## Quantization Details

- Source: local oMLX-compatible `gpt-oss-120b` MLX checkpoint, originally derived from `openai/gpt-oss-120b`.
- Quantizer: oMLX oQ4.
- Main quantized tensors: affine oQ4, `bits: 4`, `group_size: 64`.
- GPT-OSS MoE expert projections: MXFP4 passthrough, `bits: 4`, `group_size: 32`, `mode: mxfp4`.
- Expert override coverage: all 36 layers for `gate_proj`, `up_proj`, and `down_proj` under `model.layers.<i>.mlp.experts`.
- Floating dtype used during quantization: `bfloat16`.

The MXFP4 expert overrides are required for oMLX/MLX loading. Without them, the loader treats the expert tensors as affine-quantized tensors and expects `*.biases` parameters that do not exist for these MXFP4 expert weights.

## Use With oMLX

Download into an oMLX-discoverable model directory:

```bash
hf download cjnielson44/gpt-oss-120b-oQ4 \
  --local-dir ~/.omlx/models/cjnielson44/gpt-oss-120b-oQ4
```

Restart oMLX, then use this model id:

```text
gpt-oss-120b-oQ4
```

Example OpenAI-compatible request, assuming your oMLX server is listening locally:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OMLX_API_KEY" \
  -d '{
    "model": "gpt-oss-120b-oQ4",
    "messages": [{"role": "user", "content": "Write a short note about Apple Silicon inference."}],
    "max_tokens": 128
  }'
```

## Choosing This Variant

Use oQ4 if you want the smallest non-expert tensor precision among these releases. Because GPT-OSS expert tensors are preserved as MXFP4 in all three variants, the practical disk-size difference between oQ4, oQ6, and oQ8 may be smaller than expected.

For best quality from this release family, prefer `cjnielson44/gpt-oss-120b-oQ8`.

## Verification

This repo was uploaded after local oMLX discovery and load/unload smoke testing. The same GPT-OSS MXFP4 expert override fix used for oQ8 was applied to this oQ4 repo.

## Limitations

- Experimental community quantization.
- Requires recent oMLX/MLX support for GPT-OSS and MXFP4 expert tensors.
- No benchmark or perplexity numbers are provided yet.
- This model card does not change the upstream license or usage terms of `openai/gpt-oss-120b`.