KaedeTai/Ornith-1.5-35B-A3B-BigBang-MTP-mlx-4bit

Ornith-1.5-35B-A3B-BigBang-MTP — MLX 4-bit, with the MTP head and the vision tower intact

MLX 4-bit conversion of EryriLabs/Ornith-1.5-35B-A3B-BigBang-MTP. 67 GB bf16 → 19.4 GB, 2134 tensors.

Two things survive here that the obvious conversion route destroys:

	this repo	mlx_lm.convert	other MLX 4-bit builds of this family
MTP head (44 tensors)	yes	silently dropped	no
Vision tower (333 tensors)	yes	—	dropped

Why the default route loses the head

mlx_lm.models.qwen3_5.TextModel.sanitize contains:

weights = {k: v for k, v in weights.items() if "mtp." not in k}

Every mtp.* tensor is removed during conversion. No warning. For a checkpoint whose entire reason to exist is its MTP head, the output is the base model with extra steps — and it still looks fine, because the base model works.

oMLX's own error text documents the same thing from the other side: "Default mlx-lm converters strip them; you need a converter that preserves MTP."

Four traps, and how each was verified

Nothing below was assumed. Each was checked numerically against a conversion of the same architecture that already loads.

1. Fused expert weights. HF stores mlp.experts.gate_up_proj as [E, 2I, H]; MLX wants mlp.switch_mlp.gate_proj/up_proj as [E, I, H]. The split is by block, not interleaved — block scores 0.12 mean relative difference against the reference conversion (i.e. 4-bit quantisation noise), interleaved scores 1.63.

2. Zero-centred norms. RMSNorm weights need +1. With the shift, 0.001 relative against the reference; without, ~1.0. The gated norm inside linear_attn is not zero-centred and must be left alone.

3. linear_attn.conv1d.weight is [C, 1, K] in HF, [C, K, 1] in MLX.

4. vision_tower.patch_embed.proj.weight is [O, C, T, H, W] in HF, [O, T, H, W, C] in MLX. This one is worth dwelling on. Get it wrong and mlx_vlm still works perfectly — its own sanitize fixes the axes at load time, so an image test passes and you conclude vision is fine. oMLX loads strictly, fails, and falls back to serving the model text-only, leaving one WARNING line in a log nobody reads:

VLM loading failed ..., falling back to LLM:
  Expected shape (1152, 2, 16, 16, 3) but received shape (1152, 3, 2, 16, 16)
Successfully loaded ... as LLM (fallback from VLM)

This build was verified by replicating oMLX's strict load path (no sanitize): 0 shape mismatches, 0 missing parameters — identical to a checkpoint oMLX is known to serve with vision.

Measured

Production, oMLX, 659 real requests (M5 Max 128 GB, macOS 26.4.1):

depth-1 acceptance	63.7% (10843/17032)
depth-2 acceptance	64.8% (4707/7266)
depth-3 acceptance	63.5% (1394/2195)
tokens per backbone forward	median 1.91, mean 2.05, max 5.00
MTP head share of compute	4.6% (backbone 853 s vs head 42 s)
requests where the controller parked MTP	72/659 (11%)

Acceptance does not decay with depth here — depth 2 actually scores higher than depth 1. Dense Qwen3.8-27B heads fall off sharply over the same range (98 / 91 / 85%). A flat curve means depth is cheap to buy — and it is: raising mtp_num_draft_tokens from 3 to 4 is the one optimisation here that paid (§Set the draft depth to 4). oMLX's controller caps at 8.

The distribution is the honest version of "how much faster". Half of all requests emit under 1.91 tokens per forward; the tail reaches 5.00. The 11% at exactly 1.00 are requests where oMLX measured that speculation was not paying and parked it — which is the behaviour you want unattended, and the reason a bad head costs speed rather than correctness.

The head is an entire MoE layer — 256 experts plus a shared expert — which sounds expensive and is not. Drafting runs one token, where top-8 routing is cheap, against a 40-layer backbone. A typical request reads timing[backbone=308.0ms mtp=9.5ms].

Cross-stack agreement. EryriLabs measured 0.55–0.75 acceptance with llama.cpp speculative decoding on an RTX 3090. This build measures 61.5% under MLX on Apple Silicon. Two unrelated stacks, same answer — acceptance is a property of the weights, not the runtime.

Model-level acceptance (teacher-forced against the trunk's own argmax, no server, no thermals; post_norm hidden variant):

distribution	acceptance
code	81.5%
Chinese — the model's own replies	71.6%
English prose	70.0%
Chinese — web corpus (CC-100 zh-Hant)	56.8%
Chinese — news corpus (Wikinews)	56.5%

The two Chinese rows differ by 15 points, and only the top one is real. An earlier revision of this card reported 55–57% and called Chinese "40 points behind code, the obvious target for a head fine-tune". That number came from web and news corpora — text this model never generates. On what it actually writes, Chinese is level with English and 10 points behind code.

The row above is still not the last word: those replies answer prompts written to elicit them. Measured on one real deployment's traffic — the questions people actually asked, under that product's system prompts — the same stock head accepts 73.5% on long-form health consultation and 64.0% on short spoken customer-service replies. Register moves this more than language does.

That mistake cost two training runs; §Fine-tuning below is what they showed.

Does an image in the context hurt acceptance?

The head never sees image features. The question is whether the text it drafts stays predictable with ~320 image tokens sitting in the context. Four matched pairs — the same request with a real image, and with the image replaced by a written description — 368 drafted positions per depth:

It costs about 5 points at draft 1 and nothing by draft 3. Leaving MTP on for image traffic is worth it. (That means mtp_enabled alone — see the warning about vlm_mtp_enabled below.)

Throughput

512-token generations on an idle M5 Max, 3 interleaved rounds, timed from the client. The AR reference is Ornith-1.5-35B-A3B-MLX-4bit — same architecture, same 4-bit g64 quantisation, zero mtp.* tensors.

	Chinese			English
	tok/s	×AR	runs	tok/s	×AR	runs
AR (no MTP head)	122.7	1.00×	132, 123, 120	135.5	1.00×	132, 136, 139
MTP, stock head	143.3	1.17×	145, 139, 143	156.6	1.16×	148, 157, 158
MTP, Chinese-tuned head	137.6	1.12×	138, 136, 143	157.0	1.16×	150, 158, 157

Medians of 3 interleaved rounds; every individual run is listed. Do not read the third row as the tuned head being slower on Chinese — with n=3 on a machine that drifts 8% with thermals, a 4% difference is not resolvable. A paired benchmark (the two heads run back to back on the same prompt, 4 Chinese tasks × 3 rounds) puts them at a ratio of 1.007, the tuned head winning 9 of 12 pairs — consistent in direction, ~1% in size, not significant.

Three numbers that matter more than the headline:

• Only measure this on an idle machine. Anything else sharing the GPU moves the number by an amount set by that other work, not by the model — so a tok/s figure taken under unspecified load says nothing that can be reproduced.

• oMLX's own per-request tok/s field is not usable, and neither is the dashboard average built from it. Across 274 logged requests on this model:

  generated	oMLX reports	actual tokens / wall-clock
  10–30 tok	365.3 tok/s	40.0
  30–60	161.0	52.5
  60–120	135.6	72.1

  The reported figure falls as generations get longer while the true rate rises — the signature of a fixed per-request cost divided by a small decode time. Short replies dominate ordinary traffic, so an average over them reads far above anything a long generation reproduces. Time it from the client.

• Content matters more than the head. Chinese narrative prose runs ~127 tok/s where Chinese technical writing runs ~165 — a bigger spread than anything the draft head changes.

Vision and MTP run together

The engine log for a healthy load reads:

VLMBatchedEngine loaded: .../Ornith-BigBang-MTP-mlx-4bit
Speculative backend selected: Lightning MTP (model_type=qwen3_5_moe, active)
MTP[1] tokens=512 cycles=228 tok/cycle=2.25 accept=282/414 (68.1%)
        timing[backbone=2807.2ms mtp=121.2ms]

Do not set vlm_mtp_enabled. Despite the name it is a different feature — speculative decoding via an external assistant drafter, targeting a Gemma4 body — and oMLX treats it as mutually exclusive with mtp_enabled. Setting both silently disables the built-in head: the model loads, serves, and looks fine, with no Lightning MTP line in the log. mtp_enabled: true alone is what turns this head on, on the VLM path included.

Do not benchmark this under MTPLX

MTPLX 2.9.1 reports 5.64% acceptance and runs 0.34× of no-MTP on this artifact. That is MTPLX mis-reading a MoE expert stack as a dense layer, not a property of the model — the same weights give 61.5% in production under oMLX and 0.55–0.75 under llama.cpp. It does not error; it just returns a number that looks exactly like "MoE is unsuited to speculative decoding". It is not.

Set the draft depth to 4

The one optimisation here that paid, and it needs no training:

mtp_num_draft_tokens defaults to 3 for this family. Raising it to 4:

content	depth 4 vs depth 3	rounds won
Chinese, technical	+10.8%	6/7
code	+4.5%	4/7
Chinese, procedural	+1.3%	6/7
English, technical	+0.9%	4/7
Chinese, narrative	−5.0%	0/7

Paired, 35 comparisons, identical weights — only the setting differs. The per-depth acceptance explains it: d4 accepts 71.1%, higher than d2 (64.0%) or d3 (68.0%); d5 collapses to 42.9%. Depth 8 is worse than 3 overall (0.972×) because the controller spends cycles probing d5–d8.

Narrative prose loses because low-acceptance text wastes a deeper draft. If your traffic is technical, procedural or code — the usual case for an assistant — depth 4 is the better default. The adaptive controller is supposed to back off per sequence and does not do so quickly enough on narrative.

One tensor, one file

If you build a variant by hard-linking the trunk shards and writing a new head to a separate file, the head will exist twice: it also lives inside one of those shards. Loaders glob *.safetensors and weights.update(), so whichever sorts last wins — silently, with no error, when the two copies have matching shapes. swap_head.py in this repo rewrites the shard that holds the head and asserts afterwards that no tensor name appears in two files.

Usage (oMLX)

Set mtp_enabled: true in ~/.omlx/model_settings.json — that one flag, on the vision path included, and not vlm_mtp_enabled. Then unload and reload; oMLX reads model settings at load time only. Expect:

Speculative backend selected: Lightning MTP (model_type=qwen3_5_moe, active)

If you instead see falling back to LLM, the vision axes are wrong in whatever build you are using.

Size

trunk	18.169 GiB
MTP head (44 tensors)	0.454 GiB
vision tower (333 tensors, dense bf16)	0.832 GiB

Quantisation follows oMLX's own predicate: 4-bit g64 affine, 8-bit for mlp.gate and shared_expert_gate (routers are error-sensitive), mtp.fc left dense, vision left dense.

Limitations

• One machine, one MLX version (0.32.0 / mlx-vlm 0.6.13).
• Vision + MTP is measured on 4 matched pairs (368 positions/depth) — enough to rule out a large effect, not enough to pin down a small one.
• No long-context or concurrency numbers. All figures are single-request; the production sample averages ~50 generated tokens per request.
• The depth sweep is 35 paired comparisons on one machine. Depths 5–8 were measured only in aggregate (depth 8 is 0.972× of depth 3); the per-content breakdown exists for 3 vs 4 only.
• A Chinese-tuned head is available at KaedeTai/Ornith-1.5-35B-A3B-BigBang-MTP-zh-mlx-4bit. Two attempts there produced nothing; the third accepts 3.41 points more of its own drafts by training on one deployment's real traffic, masking the loss to assistant positions, and freezing the expert stack. That is draft accuracy, not throughput — see that card for what is and is not measured.

Credits

• EryriLabs — the merge, the grafted head, and the random-init diagnostic that started it
• ornith-ai · endless-frontier · Qwen
• oMLX — Lightning MTP, and the adaptive controller that parks itself when speculation stops paying

Converter: hf_moe_to_mlx.py in this repo, also at KaedeTai/Qwen3.x-27B-mlx-mtp-graft.