Capicua25x/Ornith-1.0-35B-MXFP4-Vision-MTP

Ornith-1.0-35B-A3B — MXFP4 + MTP (vision), for AMD RDNA4 / vLLM

deepreinforce-ai/Ornith-1.0-35B — a Qwen MoE model (A3B, ~3 B active parameters per token) — quantized to MXFP4 with a grafted MTP (Multi-Token-Prediction) draft head, packaged to run out of the box on AMD Radeon RDNA4 (gfx1201) under vLLM with lossless self-speculative decoding. Vision retained.

• Trunk — Ornith-1.0-35B quantized to MXFP4 (compressed-tensors, group size 32, symmetric). lm_head, embed_tokens, norms, and the vision tower are kept BF16. The MXFP4 trunk is byte-verified against DeepReinforce's published weights (16/16 shards SHA256-identical before quant).
• Draft head — a cross-model graft. The MoE MTP head (785 BF16 tensors, with per-expert weights) is transplanted from a sibling Qwen-MoE checkpoint, Capicua25x/Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill-MXFP4-Vision (a DeepSeek-V4-Pro-Thinking distill of Qwen3.6-35B-A3B). Because the trunk and the donor share the same Qwen MoE architecture — identical hidden size and expert layout — the head transfers cleanly and drafts well on the Ornith trunk (acceptance below).
• Speculative decoding — vLLM native mtp method, num_speculative_tokens=3. Lossless: the target model verifies every drafted token, so the output distribution is identical to the trunk with no draft head. A mismatched head could only slow drafting, never change an output — which is what makes a cross-model graft safe to ship. This one drafts at ~69% acceptance, a clear net speed-up.

Measured on AMD RDNA4

2× Radeon AI PRO R9700 (gfx1201), tensor-parallel 2, vLLM 0.19.1 (image below).

• MTP draft acceptance (n=3): ≈69.5% average — measured on live production traffic, not a synthetic probe. Per-position 0.84 / 0.67 / 0.57; mean acceptance length ~3.08. Lossless throughout.
• Vision, tool-calling (qwen3_xml), and reasoning split (qwen3) all confirmed working.
• Throughput (TP2, 256 output tokens, per-user / aggregate tok/s):

concurrent	short prompt	~6k prompt
1	130.9 / 131	100.9 / 101
16	58.9 / 889	34.2 / 523
32	40.2 / 1219	20.3 / 617
64	29.9 / 1822	12.3 / 723
96	23.4 / 1584	9.2 / 717
128	21.4 / 1779	8.2 / 761

Usable concurrency ceiling (per-user ≥ 20 tok/s): ~128 at short context, ~32 at 6k context. Single-stream decode is 130.9 tok/s short / 100.9 tok/s at 6k — the A3B sparsity (~3 B active) makes it faster single-stream than a dense 9B at short context, and comparable at long context, while carrying far more total knowledge.

Run it on AMD RDNA4

Uses the prebuilt RDNA4 vLLM image capicua25x/vllm-rocm-rdna4 (tag 0.19.1):

docker run --rm --network=host \
  --device=/dev/kfd --device=/dev/dri \
  --group-add=video --group-add=render --ipc=host --ulimit memlock=-1 \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  capicua25x/vllm-rocm-rdna4:0.19.1 \
  --model Capicua25x/Ornith-1.0-35B-MXFP4-Vision-MTP \
  --served-model-name ornith --trust-remote-code \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 262144 \
  --attention-backend TRITON_ATTN \
  --enable-prefix-caching --max-num-seqs 64 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

The settings that actually matter on RDNA4

• --attention-backend TRITON_ATTN — required on gfx1201; the 0.19.1 default (ROCM_ATTN) collapses spec-decode throughput at concurrency.
• --speculative-config '{"method":"mtp","num_speculative_tokens":3}' — enables the grafted MTP head. n=3 maximizes throughput; n=1–2 maximize per-token acceptance. Tune per workload.
• --tool-call-parser qwen3_xml --reasoning-parser qwen3 — Qwen-family tool-calling + reasoning split.
• --trust-remote-code — the Qwen MoE vision architecture.
• Memory — fits 2× 32 GB at MXFP4 with 256k context (--gpu-memory-utilization 0.92, --max-num-seqs 64). The 35 B does not fit a single 32 GB GPU at this context; use TP2.

How it was built (reproducible)

1. MXFP4 quantize Ornith-1.0-35B with compressed-tensors (4-bit float, group 32, symmetric; lm_head / embed_tokens / norms / vision tower left BF16).
2. Graft the mtp.* MoE head (785 BF16 tensors — including the per-expert mtp.layers.0.mlp.experts.N.* projections) from the donor checkpoint into a new model-mtp.safetensors shard and patch model.safetensors.index.json.
3. Mark the head unquantized — add every mtp.* Linear module (the fc, the attention projections, and all per-expert MLP projections) to quantization_config.ignore, so vLLM's compressed-tensors loader keeps the BF16 head as-is instead of expecting MXFP4 weight-scales. This is the one mixed-precision gotcha; for a MoE head it means hundreds of ignore entries (derived automatically from the head's rank-2 tensors).

Step 1 (the MXFP4 quantize) is scripted in quantize_mxfp4.py (weight-only MXFP4A16, group 32, data-free — deterministic/byte-reproducible). Steps 2–3 are scripted in recipe_graft_mxfp4.py — it copies every mtp.* tensor from the donor and derives the ignore list from the rank-2 (matmul) weights, so the same script handles both a dense head and this MoE head.

Credits

• DeepReinforce — Ornith-1.0-35B, the base/trunk model (MIT).
• nerkyor — Qwen3.6-35B-A3B-DSV4Pro-Thinking-Distill, which trained the MoE MTP head grafted here.
• DeepSeek — DeepSeek-V4-Pro-Thinking, the model that distill learned from.
• Qwen / Alibaba — the Qwen MoE (A3B) architecture shared by both the trunk and the grafted head.
• vLLM and compressed-tensors — serving stack and quantization format.
• Rob Smith (tcclaviger) — the RDNA4 vLLM base image that made gfx1201 serving possible. The capicua25x/vllm-rocm-rdna4 image this model runs on is a forward-port of his work — without it, none of this runs.

License

Apache-2.0. This combines the Ornith-1.0-35B trunk (MIT, DeepReinforce) with a grafted MTP head (Apache-2.0). MIT is one-way compatible with Apache-2.0, so the combined derivative is distributed under Apache-2.0; both upstream notices are preserved above in Credits.