Muse-Glimmer-30B-NVFP4-GPTQ-mm

NVFP4 (4-bit) quantization of Muse-Glimmer-30B, calibrated on a mixed text+image set. 55.49 GB → 23.41 GB (2.37×), and on our harness it holds MMMU-val to within 0.44 points of the bf16 original.

The -mm suffix means multimodal-calibrated, and it is the whole point of this repo. We built a text-only-calibrated sibling from the identical recipe; it scored 1.89 points lower on MMMU and we did not publish it. The suffix is kept so the two builds stay distinguishable in anyone's model registry, including ours.

The point of this checkpoint is the calibration data, not the recipe. We ran the identical GPTQ NVFP4 recipe twice on the same model and changed only what we calibrated on:

calibration	MMMU-val	ChartQA (relaxed)
bf16 original (no quantization)	0.4922	0.4352
text + image (this model)	0.4878	0.4148
text only	0.4689	0.4028

Feeding images into the calibration set moved MMMU by 1.89 points with the bit width, the recipe, and the model held constant. If you are quantizing a vision-language model to 4 bits and calibrating on a text corpus, that gap is probably what you are paying.

Measured results

B200 ×1, vLLM 0.26.1rc1.dev608, lm-eval 0.4.12, full sets (MMMU-val n=900, ChartQA n=2500), n=2 reruns per candidate.

model	size	MMMU-val	ChartQA (relaxed)
Muse-Glimmer-30B (bf16)	55.49 GB	0.4922	0.4352
this model	23.41 GB	0.4878	0.4148
RedHatAI/Muse-Glimmer-30B-NVFP4	21.80 GB	0.4800	0.4048
text-only-calibrated NVFP4 (ours)	23.41 GB	0.4689	0.4028

Run-to-run spread across the n=2 pairs was 0.00–0.36 points, so the margin over the reference NVFP4 build (+0.78 MMMU, +1.00 ChartQA) sits above the measured noise floor rather than inside it.

⛔ Do not cite the ChartQA numbers as absolute performance

In the same runs, ChartQA reports exact_match 0.053 against relaxed 0.405 and anywhere 0.414. A metric collapsing to one eighth of its siblings is answer extraction failing, not the model failing: Muse-Glimmer is a reasoning model and emits its chain of thought around the answer, which exact_match cannot strip.

The relaxed and anywhere metrics agree within 0.9 points, so these numbers are sound for comparing candidates measured on the same harness — which is all we use them for above. They are not a statement of how well this model reads charts.

Quantization details

• Method: GPTQ, NVFP4, via llm-compressor oneshot
• Calibration: 1,024 samples, 512 of them carrying images
• Runtime: 3,409 s of oneshot on a single B200
• Non-expert layers follow the base recipe; see recipe.yaml in the repo

Usage

Serve with vLLM (NVFP4 needs a Blackwell-class GPU):

vllm serve ThakiCloud/Muse-Glimmer-30B-NVFP4-GPTQ-mm \
  --max-model-len 8192 --limit-mm-per-prompt '{"image": 8}'

Reproducing the numbers

lm_eval --model vllm-vlm \
  --model_args pretrained=ThakiCloud/Muse-Glimmer-30B-NVFP4-GPTQ-mm,max_model_len=8192,max_images=8 \
  --tasks mmmu_val,chartqa --apply_chat_template --batch_size auto

--apply_chat_template is not optional here. Without it this model is scored outside the template it was trained for, and the resulting numbers are not comparable to anything above.

A note on which GPU you measure this on

Everything above was measured on a single B200. That matters more than it sounds.

FP4 tensor cores arrived with Blackwell; Hopper (H200/H100) does not have them. An NVFP4 checkpoint still loads on Hopper, but the engine dequantises the weights and computes at higher precision — so a benchmark run there measures a dequantisation path, not FP4 arithmetic, and the two do not produce the same numbers. On a sibling text model we measured the same checkpoint on both: the gap to BF16 on MMLU moved from 0.36 to 1.36 points, about four times the harness standard error, purely from the hardware change.

Be careful with our numbers too. The runs above predate the kernel-path gate we now enforce, and their serving logs are gone, so we cannot retroactively certify which kernel they took. All four builds were measured on the same engine and the same GPU, so the comparison between them stands — that is what this repo is about. The absolute figures, including the 0.44-point gap to bf16, are conditional on that path.

Limitations

• Evaluated on MMMU-val and ChartQA only. No safety, OCR, long-context, or multilingual evaluation was run — do not assume parity with bf16 outside the two benchmarks above.
• Single quantization seed. We measured evaluation noise (n=2 reruns) but not quantization noise (a second calibration draw).
• ChartQA is 2.04 points below bf16. MMMU is essentially at parity; chart reading is not.