Ttimms/zaya1-8b-nvfp4-w4a4-uniform

ZAYA1-8B NVFP4 W4A4 — uniform (no BF16 exemptions)

6.02 GB. Every one of 1,320 Linears in packed NVFP4 W4A4 — 4-bit weights and 4-bit activations — with zero BF16 exemptions, served on a 16 GB consumer Blackwell card (RTX 5070 Ti, SM120) via native CUTLASS FP4 tensor-core kernels.

This is the companion to Ttimms/zaya1-8b-nvfp4-w4a4 (9.46 GB), which keeps 384 outlier-sensitive Linears at BF16. Both are maintained. Neither supersedes the other — the choice is 3.44 GB of VRAM against a small, measured accuracy cost.

Which one should you use?

	this checkpoint	9.46 GB variant
Weights (model.safetensors)	5.99 GB	9.46 GB
Repository total	6.02 GB	9.49 GB
BF16-exempted Linears	0	384
W4A4 Linears	1,320	936
HellaSwag acc (n=10,042)	45.79%	46.49%
HellaSwag acc_norm	60.65%	61.34%
KV cache headroom on 16 GB	6.83 GiB / ~336k tokens	materially less

Take this one if you want maximum context/KV headroom on a 16 GB card, or you are batching. The accuracy cost is −0.71 pp on HellaSwag.

Take the 9.46 GB one if you want the best measured accuracy and can afford the VRAM, or if you need the exact artifact used as the control in the evaluation below.

On the two size figures. 6.02 GB is the uniform build's whole repository; 9.46 GB is the mixed build's model.safetensors alone (its repository total is 9.49 GB). Both figures are individually correct and both are already published, but they measure different things — so the table above gives each basis explicitly. The reduction holds either way: −36.7% on weights, −36.6% on repository totals. Measured from the Hub, 2026-08-09.

⚠️ Known issue: CUDA graph capture corrupts output on SM120

Throughput figures previously published on this card (104.7 tok/s here / 407.4 tok/s batch-8 elsewhere in this project) were measured with CUDA graphs enabled — a code path confirmed 2026-08-14 to produce numerically wrong output on this hardware, independent of MoE backend. A sweep of flashinfer_cutlass (default), cutlass, and marlin (weight-only) all produced garbage under graph capture; only enforce_eager=True generated correctly. Since Marlin barely touches the FP4 MoE path and still failed, the fault is graph capture itself, not any one kernel. Two adjacent upstream issues exist — CUTLASS #3096 (different, non-graph-capture bug) and FlashInfer #2776 (graph-capture-specific, but its stated root cause doesn't explain Marlin failing too) — neither currently offers a fix that preserves CUDA graphs for this failure mode. See Pape, Evertz & Schönherr (arXiv:2605.19537) for the general phenomenon of backend-dependent correctness drift in LLM serving. Full sweep, coherence re-verification, and citations in RESEARCH.md and ROADMAP.md on the GitHub repo.

Set --enforce-eager (or enforce_eager=True) when serving this checkpoint. It is the only configuration confirmed to produce coherent output.

Throughput and memory (measured, enforce_eager=True)

Measured with vLLM's own benchmark CLI, not a bespoke harness. 5 separate process invocations per configuration, GPU otherwise idle:

vllm bench latency --model <checkpoint> --dtype bfloat16 --kv-cache-dtype fp8 \
  --gpu-memory-utilization <frac> --max-model-len 4096 --no-enable-prefix-caching \
  --input-len 128 --output-len 256 --batch-size <1|8> --enforce-eager \
  --num-iters-warmup 1 --num-iters 1

	6.02 GB uniform	9.46 GB mixed
Single-stream, median (range)	9.52 (9.48–9.84) tok/s	9.51 (9.45–9.81) tok/s
Batch-8, median (range)	73.4 (72.2–74.9) tok/s	74.4 (72.8–75.7) tok/s
Batch-8 scaling vs. batch-1	7.71× (96% of ideal)	7.82× (98% of ideal)
--gpu-memory-utilization	0.85	0.92

Throughput is still unchanged between checkpoints — the two land within run-to-run noise of each other at both batch sizes. That part of the original claim held; only the absolute numbers were wrong.

Batching is near-ideal (96–98% of theoretical linear scaling on batch-8), which is evidence the MoE decode step's per-step cost does not grow meaningfully with batch size.

The two checkpoints cannot be run at the same memory fraction on a 16 GB card. The mixed checkpoint fails at --gpu-memory-utilization 0.85 with ValueError: No available memory for the cache blocks, and at 1.0 with Free memory on device cuda:0 (14.66/15.92 GiB) ... less than desired because a desktop session holds ~1.3 GiB. Its working range is narrow; the uniform build runs at 0.85 with headroom — the practical case for the smaller checkpoint, more so than the accuracy difference.

Retracted, do not cite: 104.7 / 105.3 tok/s single-stream, 102.6 / 407.4 tok/s from elsewhere in this project (all measured under CUDA graphs).

An unverified external signal exists and disagrees with the table above. llama.cpp PR #23112's own author reports 45.9 tok/s on a slower RTX 4070 Ti (Q4_K_M GGUF) — beating this checkpoint's 9.5 tok/s by ~4.8×. Five attempts to reproduce it on this project's own SM120 hardware hit the same non-deterministic hang each time, ruling out the model and toolkit version as causes; looks like a WSL2/driver-level issue. Not resolved as of 2026-08-14 — see GitHub repo RESEARCH.md §5.16 for the full diagnostic log.

Update, same day: the gap likely has a principled cause independent of the hang above. Activation quantization (W4A4) gives no speed benefit at batch-1 — decode there is memory-bandwidth-bound, and quantizing activations only helps when compute is the bottleneck. Weight-only quantization is expected to win at batch-1 by design; this checkpoint's advantage is memory footprint and batched throughput (see the batch-8 row above). Detail: RESEARCH.md §5.17.

A real batch-1 lever that does exist: vLLM's built-in n-gram speculative decoding gives a validated 2.2× speedup on coding-edit prompts (zero training, one config flag) — no gain on free-form generation, as expected. Deployed to the production serve script and validated live through the real OpenAI-compatible API. Detail: RESEARCH.md §5.18.

Accuracy: measured, paired, and honest about its limits

The two checkpoints were compared with a paired exact-binomial McNemar test on discordant items, joined per doc_id, over 14,319 items each across four pure-loglikelihood tasks. No generation is involved, so this is immune to the <think>-never-terminates artifact that invalidates naive generative evals on this model. No chat template — these are ranked-continuation tasks.

task	metric	n	this (6.02 GB)	control (9.46 GB)	Δ pp	95% CI	p
hellaswag	acc	10,042	45.79%	46.49%	−0.71	[−1.26, −0.15]	0.0140
hellaswag	acc_norm	10,042	60.65%	61.34%	−0.70	[−1.39, −0.01]	0.0504
arc_challenge	acc	1,172	37.97%	36.95%	+1.02	[−1.42, +3.47]	0.4522
arc_challenge	acc_norm	1,172	37.97%	40.36%	−2.39	[−4.97, +0.19]	0.0799
winogrande	acc	1,267	56.20%	59.04%	−2.84	[−6.04, +0.36]	0.0906
piqa	acc	1,838	69.42%	70.02%	−0.60	[−2.41, +1.21]	0.5564
piqa	acc_norm	1,838	70.89%	70.08%	+0.82	[−1.02, +2.65]	0.4166

Read the intervals, not the p-values. No comparison survives Bonferroni correction (α = 0.05/7 = 0.0071). That is absence of resolution, not evidence of absence. HellaSwag is the only adequately powered task here, and its 95% CI excludes zero. The three smaller benchmarks cannot rule out regressions of −4.97 pp (arc_challenge acc_norm) or −6.04 pp (winogrande). Five of seven comparisons point negative, consistent with a small real regression that only HellaSwag has the samples to detect.

Defensible claim: −0.71 pp on HellaSwag for −36% checkpoint size. Anything stronger is unsupported by this data.

An earlier ARC-Easy comparison suggested no cost at all. That was an unpaired two-proportion test applied to paired data: both checkpoints are quantizations of one base model scored on the same items, and discarding that correlation discards the statistical power. It also happened to favour the wrong checkpoint. Aggregate accuracy output cannot be converted into a paired test after the fact — it requires re-running the model with per-item logging.

Reproduce with scripts/run_phase_a.py and scripts/analyze_phase_a.py in the GitHub repo.

Generative benchmarks (2026-08-16)

The tests above are loglikelihood — they score pre-written continuations, so they measure ranking, never producing. These three measure generation:

benchmark	score	95% CI	n
HumanEval	72.6% pass@1	[65.3, 78.8]	164
GSM8K	65.5%	[62.9, 68.0]	1,319
MMLU-Pro (0-shot)	48.1%	[44.5, 51.8]	700

enforce_eager=True, think_budget 4096, temp 0.6 / top_p 0.95 (Zyphra's published recommendation), seed 42. Reproduce with scripts/run_budget_forced_suite.sh.

On HumanEval: published figures put Qwen 3 7B at roughly 68–72% and Llama 3 8B at 62–65% — both at full precision. This checkpoint matches or exceeds them at 4-bit weights and activations in 6.02 GB. Zyphra publishes no HumanEval number for ZAYA1, so there is no retention figure to quote; this appears to be the first measurement of it at any precision.

On MMLU-Pro — read this before comparing to Zyphra's 74.2%. That number is not a valid comparison point for this one. lm-eval's MMLU-Pro task is 5-shot; this harness is 0-shot, because ZAYA1's <think> trace frequently never terminates and the standard harness scores the unterminated trace as the answer (the artifact that put IFEval at 19.8% against an 85.58% reference). Zyphra's figure also comes from a private harness with undisclosed generation limits. The gap is a protocol difference, not quantization damage: published INT4 loss on MMLU-Pro is ~1.6 pp, and the paired test above bounds this checkpoint's cost at −0.71 pp HellaSwag.

The budget was tested, not assumed. Both benchmarks were re-run at think_budget 8192 and compared with paired McNemar on identical items — GSM8K +0.15 pp (p=0.9581), MMLU-Pro +3.29 pp (p=0.0673). Neither is significant, so 4096 is the correct operating budget and more reasoning room does not help this model. Detail: RESEARCH.md §5.22.

On GSM8K: honest but unremarkable — Llama 3.2 3B reports 77.7% at 8-shot. This run is 0-shot, 78% of traces truncate, and GSM8K is heavily contaminated. Treat it as a regression check that quantization did not break arithmetic reasoning, not as evidence of mathematical ability.

⚡ enable_thinking=False — 8.5× faster, but it costs 17–29 accuracy points

ZAYA1's chat template (Zyphra's, not ours) supports disabling the reasoning trace, which pre-closes <think> so the model answers immediately. This is the single biggest latency lever available, and it is not free. Measured on this checkpoint, paired McNemar on identical items:

benchmark	thinking	enable_thinking=False	Δ	p	wall time
HumanEval	72.6%	43.9%	−28.66 pp	<0.0001	15 m → 2 m
MMLU-Pro	48.1%	26.7%	−21.43 pp	<0.0001	39 m → 4 m
GSM8K	65.5%	48.1%	−17.36 pp	<0.0001	63 m → 8 m

Read this before deciding how to serve the model: ZAYA1's accuracy is its reasoning, and its reasoning is what makes it slow — the two cannot be separated. It is not a fast interactive model, and no serving configuration makes it one. If you need low latency more than accuracy, a weight-only quant of a non-reasoning model will serve you better.

Where the flag is genuinely useful: per-request routing, not a global switch. vLLM accepts it per request:

{"model": "...", "messages": [...],
 "chat_template_kwargs": {"enable_thinking": false}}

Send mechanical work (file reads, trivial edits, simple tool calls) down the fast path and keep reasoning for real problem-solving. Note that 12 / 53 / 173 items respectively were solved only with thinking disabled — some tasks are actively hurt by overthinking, so routing can win twice.

Full analysis: RESEARCH.md §5.23.

Why the exemptions were removable

The 9.46 GB checkpoint exempts 12 MoE layers whose calibrated activation max_abs exceeds 500 — the worst being 8,896 at L75.experts.1.linear_fc2, 622× the median. Only 24 linear_fc2 modules actually offend, but FusedMoE requires uniform quantization per layer, so protecting them forces exemption of fc1 and fc2 across all 16 experts in each affected layer: 384 Linears, 3.44 GB. A 16× overpay, where 16 is num_experts.

The exemptions turned out to be largely redundant. SOAR — which targets the same FP8 block-scale rounding error the outliers cause — was introduced after the mixed-precision decision, and the combination was never re-tested. The worst channel in the model quantizes coherently at W4A4 without them.

No residual correction (ARCQuant or otherwise) is applied to this checkpoint, and none is required. The numbers above are for the artifact exactly as published.

Base model revision — read before reproducing

Quantized from the original 80-layer ZAYA1-8B config (num_hidden_layers: 80, moe_router_topk, zaya_use_eda / zaya_use_mod, transformers 4.57.1).

In late June 2026 Zyphra refactored ZAYA1-8B into upstream-transformers form: Zyphra/ZAYA1-8B now reports num_hidden_layers: 40 with layer_types: hybrid and num_experts_per_tok, and the original moved to Zyphra/ZAYA1-8B-legacy.

Core dimensions are unchanged (hidden 2048, 16 experts, top-1 routing, vocab 262272), so this is a re-expression rather than a new model — but reproduce against Zyphra/ZAYA1-8B-legacy, or you will hit an architecture mismatch.

Verified, not just inferred (2026-08-14): fetched model.embed_tokens.weight directly from both repos via HTTP range request (no full download) and compared the raw bytes — byte-for-byte identical, 1,074,266,112 bytes, BF16, shape [262272, 2048]. The two repos' safetensors index files also report an identical aggregate total_size (17,680,978,928 bytes) despite the tensor layout changing from 2,483 named tensors (legacy) to 1,283 (current, fused/batched expert tensors, transformers-conventional naming). This is why base_model lists both repos below with base_model_relation: quantized — not an assumption, a direct measurement on the one tensor that could be checked without reverse-engineering the expert-fusion mapping. Every other published NVFP4/GGUF/BNB quantization of this model tags only the current Zyphra/ZAYA1-8B — this checkpoint is the only one built from and verified against the pre-refactor structure directly.

Quantization details

Field	Value
Base model	Zyphra/ZAYA1-8B-legacy (pre-refactor 80-layer config)
Method	compressed-tensors, float-quantized
Scheme	NVFP4, w4a4 (4-bit weights, 4-bit activations)
Group size	16 (tensor-group strategy)
W4A4 Linears	1,320
BF16-exempted	0
Ignored / kept BF16	lm_head, router, norm, qkv, cca modules only
Calibration	977 samples, ARC-mix set, 1,320 Linears calibrated, SOAR enabled
Compressor	NVFP4PackedCompressor (weight_packed_dtype=uint8, weight_scale_dtype=float8_e4m3fn)
Built on	RTX 5070 Ti, 16 GB VRAM, CUDA 13.0

quantization_manifest.json carries the full machine-readable config, the exact outlier-layer list, and the evaluation block reproduced above.

Usage

Requires vLLM built from source with SM120 NVFP4 CUTLASS kernels — stock wheels do not compile them. Inference must run in bfloat16.

# vLLM built from source with: TORCH_CUDA_ARCH_LIST=12.0
vllm serve Ttimms/zaya1-8b-nvfp4-w4a4-uniform --dtype bfloat16 --enforce-eager \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 5, "prompt_lookup_min": 2}'

The exact command above is scripts/serve.sh in the repo.

Serving notes for this model family:

• enable_prefix_caching=False is required — ZAYA's CCA state is not cacheable, and vLLM defaults this to True.
• --enforce-eager is required for correct output, not just recommended — CUDA graph capture corrupts generation on this card regardless of MoE backend. See "Known issue" above before serving without it.
• --speculative-config is optional but free — lossless n-gram speculative decoding, validated 2.2× faster on coding-edit prompts (no gain on free-form generation, since there's no prompt/output overlap to exploit). Detail: RESEARCH.md §5.18.
• Set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 on a 16 GB card; the estimator otherwise reserves 3.5+ GiB and pushes the KV cache into OOM.
• Generative evals need budget forcing. ZAYA is a reasoning model whose <think> trace frequently never terminates within a normal token budget, and it answers in \boxed{} form — stock harnesses score it near random for reasons unrelated to quantization.

Why this is hard

ZAYA1-8B is an 80-layer MoE (760M active / 8.4B total) with Zyphra's CCA (compressed convolutional attention). W4A4 requires calibrating activation scales, not just weights; the compressed-tensors calibration path has a silent NaN-producing trap if you calibrate through the fake-quant nn.Linear.forward wrapper; and the NVFP4 global-scale convention (2688 / max_abs, divisor form, block scales pre-multiplied) is undocumented — getting it wrong produces silent pad-token collapse rather than an error.

Known limits

• Single-stream and batch-8 throughput are now both measured for both checkpoints with vllm bench latency under enforce_eager=True (see above): 9.52 tok/s single-stream / 73.4 tok/s batch-8 here, essentially identical to the 9.46 GB build. The earlier CUDA-graph figures (104.7/105.3 single, 102.6/407.4 elsewhere) are retracted — see "Known issue" above.
• No BF16 control eval exists — a BF16 ZAYA1-8B needs ~35 GiB and does not fit the source hardware. All accuracy claims here are relative to the 9.46 GB checkpoint, not to full precision.
• The three smaller benchmarks are underpowered, as stated above.

License

Apache 2.0 — matches the ZAYA1-8B upstream license.