cyburn/Ornith-1.0-35B-4.75Bits

Ornith-1.0-35B — PrismaQuant 4.75 bpp mixed-precision

Mixed-precision quantization of deepreinforce-ai/Ornith-1.0-35B produced by PrismaQuant — a per-Linear sensitivity-driven allocator that chooses each Linear module's format individually under a total-bit budget.

The base model is a 35B-parameter multimodal Qwen3.6-family architecture with a Mixture-of-Experts (256 experts, 8 per token) language body, a 27-layer vision tower, and multi-token-prediction (MTP) heads for speculative decoding. This quantized artifact retains the full multimodal and MTP structure.

Why "every layer refracts into a different format": a naive uniform NVFP4 either leaves disk on the table (keeping everything BF16 "to be safe") or loses quality (quantizing sensitive layers to 4-bit). PrismaQuant measures the actual Fisher-weighted MSE for every (Linear, format) pair and runs a multi-choice knapsack under a total-bit budget, so every bit lives where it buys the most likelihood.

---

At a glance

Metric	BF16 source	This artifact
Size on disk	~70 GB	~22 GB
Average bits per param	16	4.75
Multimodal (vision + text)	✓	✓
MTP speculative decoding heads	✓	✓
Loads in vLLM (stock compressed-tensors)	✓	✓
Runtime backend	any	vLLM only

---

Precision mix

This checkpoint uses three precisions, selected per-Linear by the allocator from measured sensitivity — not chosen uniformly. The visual encoder is kept entirely at BF16, preserving full visual fidelity at the cost of a small disk overhead vs a quantized visual tower.

Allocator result (text body, 401 Linears):

Format	Bits	Count (body alloc)
NVFP4	4.50	54 Linears
MXFP8_E4M3	8.25	151 Linears
BF16	16.00	196 Linears

Export recipe (all modules, 693 entries including MTP + visual passthrough):

Format	W	A	Use	Count
NVFP4	4-bit (FP4, group_size=16 with per-group FP8 scale + per-tensor global)	4-bit (dynamic)	Lower-sensitivity MoE per-expert + some dense Linears	~78 per-expert entries + 54 dense
MXFP8	8-bit (E4M3, group_size=32 with per-group E8M0 scale)	8-bit (dynamic)	High-sensitivity dense Linears the allocator won't risk at 4-bit	151 Linears
BF16	16-bit	16-bit	Router, norms, biases, embed / lm_head, pos_embed, full visual encoder, highest-sensitivity Linears	~260 passthrough + 145 bf16 dense

The allocator couples gate_up_proj / down_proj siblings so they share one per-tensor global scale (vLLM's FusedMoE requires this), and fused attention siblings (q_proj/k_proj/v_proj) share one per-tensor global scale.

Activation-aware passes applied during export

On every NVFP4 weight the exporter runs, in order:

1. GPTQ-OBS one-shot rounding — block-wise error propagation along the group-quant structure using the calibration Hessian. Closed-form, not iterative. Handles cross-column activation coupling.
2. Closed-form per-group scale sweep — for each 16-weight NVFP4 group, enumerate candidate scales, round each weight to its nearest codebook neighbor at every candidate scale, pick the configuration minimizing activation-weighted per-group MSE.

---

Which layers are quantized

Text body (DeltaNet linear-attention + dense MoE, 40 layers)

• Full attention Linears (q_proj, k_proj, v_proj, o_proj): mixed MXFP8 / BF16 per-Linear by sensitivity
• DeltaNet linear-attention Linears (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj): same — in_proj_a / in_proj_b variants were ineligible for MXFP8 due to kernel shape constraints
• MoE experts (gate_up_proj, down_proj): mixed per-expert precision — NVFP4 on lower-sensitivity experts, BF16 on the highest-sensitivity ones
• Shared expert MLP: mixed MXFP8 / BF16 per-Linear policy
• Router (mlp.gate): always BF16 (tiny, sensitive)

Multi-token-prediction (MTP) head

Speculative-decoding head (1 layer) + its own MoE block quantized under the same per-Linear policy. --speculative-config method=mtp drafts at the same precision as the body.

Visual encoder (all BF16)

All visual Linears are kept at BF16 via --visual-format=BF16. This is a deliberate conservative choice — the visual tower is left at full precision rather than running sensitivity-driven allocation on it.

Passthrough (unquantized)

• lm_head — kept at BF16. vLLM's ParallelLMHead only accepts a single weight parameter; the compressed-tensors runtime rejects a compressed lm_head. This is a vLLM runtime limitation, not a PrismaQuant design decision. The allocator would have chosen NVFP4 here.
• RMSNorm weights (all layers + MTP + visual)
• All biases
• embed_tokens
• model.visual.pos_embed

---

Serving (vLLM only)

This artifact is only runnable via vLLM's stock compressed-tensors support. vLLM 0.11+ is required.

vllm serve cyburn/Ornith-1.0-35B-4.75bits \
    --trust-remote-code \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Speculative decoding

This model supports MTP (multi-token prediction) speculative decoding. Configure via:

--speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Visual inputs

Visual inputs work via vLLM's standard image-text-to-text chat API — no special flags required.

---

Reproducing this artifact

Full pipeline is in the PrismaQuant repo:

1. Sensitivity probe — streaming per-shard empirical-Fisher trace (diagonal) across body + MTP Linears.
2. Per-(Linear, format) cost measurement — for each Linear and each candidate format, the per-group RTN error weighted by cached input activations.
3. Multi-choice knapsack allocator — picks one format per Linear minimizing total predicted Δloss under the bit budget. Target 4.75 bpp.
4. Export — streams each body / MTP shard, applies GPTQ + activation-weighted scale sweep to its NVFP4 entries, writes the compressed-tensors format. Visual encoder stamped uniformly at BF16 via --visual-format=BF16.

---

Known issues / limitations

• vLLM only at serve time. No transformers-runtime path for this precision mix today.
• lm_head stays BF16 because vLLM's ParallelLMHead does not register the NVFP4/MXFP8 compressed-tensors schemes. Costs a small amount of disk vs full NVFP4.
• Visual encoder is BF16. If disk size is the primary constraint and visual quality is not critical, a variant with sensitivity-driven visual quantization could reduce the visual tower to ~22% of its BF16 size.
• MTP n=3 is the measured optimum for this family.

---

Links

• PrismaQuant: github.com/RobTand/prismaquant
• Base model: deepreinforce-ai/Ornith-1.0-35B

---

Citation

@software{prismaquant2026,
  title        = {PrismaQuant: per-Linear sensitivity-driven mixed-precision
                  quantization for LLMs},
  author       = {Tand, Rob},
  year         = 2026,
  url          = {https://github.com/RobTand/prismaquant},
}