Ttimms/zaya1-8b-nvfp4-w4a4

ZAYA1-8B NVFP4 W4A4

NVFP4 W4A4 inference for Zyphra's ZAYA1-8B on consumer Blackwell (RTX 5070 Ti, SM120) — 4-bit weights and 4-bit activations running on native CUTLASS FP4 tensor-core kernels, at 9.51 tok/s single-stream / ~74 tok/s batch-8 (enforce_eager=True) from a 9.46 GB mixed-precision checkpoint, built and served within a 16 GB VRAM budget. An earlier 102.6/407.4 tok/s figure was measured under CUDA graphs, a path confirmed 2026-08-14 to corrupt output on this card — see "Known issue" below, retracted.

Activation quantization is the distinguishing feature here. Vendor-official 4-bit releases — Google's Gemma 4 qat-w4a16, Zyphra's own ZAYA1-8B-MXFP4-Experts — are weights-only. This checkpoint quantizes activations too, across 936 Linears.

A smaller companion checkpoint exists

Ttimms/zaya1-8b-nvfp4-w4a4-uniform is the same model with zero BF16 exemptions — every one of 1,320 Linears in packed NVFP4 W4A4, at 6.02 GB instead of 9.46 GB.

	this checkpoint	-uniform
Size	9.46 GB	6.02 GB (−36%)
BF16-exempted Linears	384	0
HellaSwag acc (n=10,042)	46.49%	45.79%
HellaSwag acc_norm	61.34%	60.65%

The 384 exemptions cost 3.44 GB and buy 0.71 pp of HellaSwag accuracy (95% CI [−1.26, −0.15], paired exact-binomial McNemar over 14,319 items). Take this checkpoint for best measured accuracy; take -uniform for KV/context headroom on a 16 GB card. Full paired table on the -uniform card.

This checkpoint is also the control for that evaluation, so it is kept unchanged.

Manifest metadata note. quantization_manifest.json in this repo records "model": "Zyphra/ZAYA1-8B". The actual base is the pre-refactor 80-layer config, now published as Zyphra/ZAYA1-8B-legacy (see below). The manifest field predates Zyphra's June 2026 refactor and is left as-is rather than rewritten, because this artifact is a published evaluation control.

⚠️ Known issue: CUDA graph capture corrupts output on SM120

The 102.6 / 407.4 tok/s figures previously published here were measured with CUDA graphs enabled — a code path confirmed 2026-08-14 to produce numerically wrong output on this card, independent of MoE backend. A sweep of flashinfer_cutlass (default), cutlass, and marlin (weight-only) all produced garbage under graph capture; only enforce_eager=True generated correctly. Since Marlin barely touches the FP4 MoE path and still failed, the fault is graph capture itself, not any one kernel. Two adjacent upstream issues exist — CUTLASS #3096 (different, non-graph-capture bug) and FlashInfer #2776 (graph-capture-specific, but its stated root cause doesn't explain Marlin failing too) — neither currently offers a fix that preserves CUDA graphs for this failure mode. See Pape, Evertz & Schönherr (arXiv:2605.19537) for the general phenomenon of backend-dependent correctness drift in LLM serving. Full sweep, coherence re-verification, and citations in RESEARCH.md and ROADMAP.md on the GitHub repo.

Set --enforce-eager (or enforce_eager=True) when serving this checkpoint. It is the only configuration confirmed to produce coherent output.

Throughput and memory (measured, enforce_eager=True)

Measured with vLLM's own benchmark CLI, not a bespoke harness. 5 separate process invocations per configuration, GPU otherwise idle:

vllm bench latency --model <checkpoint> --dtype bfloat16 --kv-cache-dtype fp8 \
  --gpu-memory-utilization <frac> --max-model-len 4096 --no-enable-prefix-caching \
  --input-len 128 --output-len 256 --batch-size <1|8> --enforce-eager \
  --num-iters-warmup 1 --num-iters 1

	6.02 GB uniform	9.46 GB mixed
Single-stream, median (range)	9.52 (9.48–9.84) tok/s	9.51 (9.45–9.81) tok/s
Batch-8, median (range)	73.4 (72.2–74.9) tok/s	74.4 (72.8–75.7) tok/s
Batch-8 scaling vs. batch-1	7.71× (96% of ideal)	7.82× (98% of ideal)
--gpu-memory-utilization	0.85	0.92

Throughput is still unchanged between checkpoints — the two land within run-to-run noise of each other at both batch sizes. That part of the original claim held; only the absolute numbers were wrong.

Batching is near-ideal (96–98% of theoretical linear scaling on batch-8), which is evidence the MoE decode step's per-step cost does not grow meaningfully with batch size.

The two checkpoints cannot be run at the same memory fraction on a 16 GB card. This checkpoint fails at --gpu-memory-utilization 0.85 with ValueError: No available memory for the cache blocks, and at 1.0 with Free memory on device cuda:0 (14.66/15.92 GiB) ... less than desired because a desktop session holds ~1.3 GiB. Its working range is narrow; the -uniform build runs at 0.85 with headroom — the practical case for the smaller checkpoint, more so than the accuracy difference.

Retracted, do not cite: 102.6 tok/s single-stream / 407.4 tok/s batch-8 (CUDA graphs; coherent output was never verified at that speed).

An unverified external signal exists and disagrees with the table above. llama.cpp PR #23112's own author reports 45.9 tok/s on a slower RTX 4070 Ti (Q4_K_M GGUF) — beating this checkpoint's 9.5 tok/s by ~4.8×. Five attempts to reproduce it on this project's own SM120 hardware hit the same non-deterministic hang each time, ruling out the model and toolkit version as causes; looks like a WSL2/driver-level issue. Not resolved as of 2026-08-14 — see GitHub repo RESEARCH.md §5.16 for the full diagnostic log.

Update, same day: the gap likely has a principled cause independent of the hang above. Activation quantization (W4A4) gives no speed benefit at batch-1 — decode there is memory-bandwidth-bound, and quantizing activations only helps when compute is the bottleneck. Weight-only quantization is expected to win at batch-1 by design; this checkpoint's advantage is memory footprint and batched throughput (see the batch-8 row above). Detail: RESEARCH.md §5.17.

A real batch-1 lever that does exist: vLLM's built-in n-gram speculative decoding gives a validated 2.2× speedup on coding-edit prompts (zero training, one config flag) — no gain on free-form generation, as expected. Deployed to the production serve script and validated live through the real OpenAI-compatible API. Detail: RESEARCH.md §5.18.

Prior art and scope of claims

switzerchees/ZAYA1-8B-NVFP4 (2026-05-19) is a genuine NVFP4 W4A4 ZAYA1-8B built with NVIDIA ModelOpt, and it predates this checkpoint by two months. Its manifest reports gpu_capability: [12, 0] — the same SM120 compute capability as an RTX 5070 Ti — so SM120 support is not a distinction of this work, and this is not the first W4A4 ZAYA1 checkpoint.

What is distinct, and all that is claimed:

Dimension	switzerchees	This checkpoint
Toolchain	NVIDIA ModelOpt v0.44.0	compressed-tensors / llm-compressor
Hardware	RTX PRO 6000, 96 GB workstation	RTX 5070 Ti, 16 GB consumer
vLLM	Zyphra prebuilt zaya1-pr	hand-built SM120 CUTLASS from source
Accuracy published	none	budget-forced GPQA-Diamond (below), plus HumanEval / GSM8K / MMLU-Pro with confidence intervals
Throughput published	none	9.5 tok/s single / ~74 tok/s batch-8, enforce_eager, median of 5 invocations. (An earlier 102.6 / 407.4 figure was measured under CUDA graphs and is retracted.)
Outlier handling	not addressed	mixed-precision exemption of 12 MoE layers

Base model revision — read before reproducing

This checkpoint was quantized from the original 80-layer ZAYA1-8B config (num_hidden_layers: 80, moe_router_topk, zaya_use_eda / zaya_use_mod, transformers 4.57.1).

In late June 2026 Zyphra refactored ZAYA1-8B into upstream-transformers form. Zyphra/ZAYA1-8B now reports num_hidden_layers: 40 with layer_types: hybrid and num_experts_per_tok, and the original was moved to Zyphra/ZAYA1-8B-legacy.

Core dimensions are unchanged (hidden 2048, 16 experts, top-1 routing, vocab 262272), so this reads as a re-expression of the same model rather than a new one — but reproduce against Zyphra/ZAYA1-8B-legacy, or you will hit an architecture mismatch.

Verified, not just inferred (2026-08-14): fetched model.embed_tokens.weight directly from both repos via HTTP range request (no full download) and compared the raw bytes — byte-for-byte identical, 1,074,266,112 bytes, BF16, shape [262272, 2048]. The two repos' safetensors index files also report an identical aggregate total_size (17,680,978,928 bytes) despite the tensor layout changing from 2,483 named tensors (legacy) to 1,283 (current, fused/batched expert tensors, transformers-conventional naming). This is why base_model lists both repos below with base_model_relation: quantized — not an assumption, a direct measurement on the one tensor that could be checked without reverse-engineering the expert-fusion mapping. Every other published NVFP4/GGUF/BNB quantization of this model tags only the current Zyphra/ZAYA1-8B — this checkpoint is the only one built from and verified against the pre-refactor structure directly.

Highlights

Result	Detail
9.51 tok/s single / ~74 tok/s batch-8	enforce_eager=True on RTX 5070 Ti — CUDA graphs produce corrupted output on this card, see "Known issue" above. Self-measured, 5 process invocations per config
9.46 GB checkpoint	936 Linears in packed NVFP4 W4A4, 384 outlier-sensitive Linears kept BF16. The BF16 exemptions are why this is larger than a uniformly-quantized export — see -uniform (6.02 GB) for that build
Checkpoint verified healthy	Budget-forced GPQA-Diamond rises monotonically with reasoning budget, 45.8% → 62.5%. See the caveat on sample size below
vLLM SM120 source build	TORCH_CUDA_ARCH_LIST=12.0 enabling cutlass_scaled_fp4_mm_sm120a + FP4 group MoE GEMM — kernels in vLLM source but not in wheels

Quantization details

Field	Value
Base model	Zyphra/ZAYA1-8B-legacy (pre-refactor 80-layer config)
Method	compressed-tensors, float-quantized
Scheme	NVFP4, w4a4 (4-bit weights, 4-bit activations)
Group size	16 (tensor-group strategy)
Ignored / kept BF16	lm_head, router, norm, qkv, cca modules, plus 12 MoE layers with max_abs > 500 (384 Linears)
Calibration	977 samples, ARC-mix calibration set, 936 Linears calibrated
Compressor	NVFP4PackedCompressor (weight_packed_dtype=uint8, weight_scale_dtype=float8_e4m3fn)
Built on	RTX 5070 Ti, 16 GB VRAM, CUDA 13.0

See quantization_manifest.json for the full machine-readable config, including the exact outlier-layer list and mixed-precision threshold.

Why this is hard

ZAYA1-8B is an 80-layer MoE (760M active / 8.4B total) with Zyphra's CCA (compressed convolutional attention) — no stock quantization path works out of the box. W4A4 requires calibrating activation scales, not just weights; the compressed-tensors calibration path has a silent NaN-producing trap if you calibrate through the fake-quant nn.Linear.forward wrapper; and the NVFP4 global-scale convention (2688 / max_abs, divisor form, block scales pre-multiplied) is undocumented — getting it wrong produces silent pad-token collapse, not an error. Full root-cause writeup in RESEARCH.md on the GitHub repo.

Usage

Requires vLLM built from source with SM120 NVFP4 CUTLASS kernels (stock wheels don't include them) — see the reproduce steps. Inference must run in bfloat16 (not fp16/fp32).

# vLLM built from source with: TORCH_CUDA_ARCH_LIST=12.0
vllm serve Ttimms/zaya1-8b-nvfp4-w4a4 --dtype bfloat16 --enforce-eager \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 5, "prompt_lookup_min": 2}'

--enforce-eager is required for correct output, not just recommended — CUDA graph capture corrupts generation on this card regardless of MoE backend. See "Known issue" above before serving without it.

--speculative-config is optional but free — lossless n-gram speculative decoding, validated 2.2× faster on coding-edit prompts (no gain on free-form generation, since there's no prompt/output overlap to exploit). Detail: RESEARCH.md §5.18. The exact command above is scripts/serve.sh in the repo.

Evaluation

Generative benchmarks were measured on the companion 6.02 GB uniform checkpoint (Ttimms/zaya1-8b-nvfp4-w4a4-uniform), not this one: HumanEval 72.6% pass@1 (95% CI [65.3, 78.8]), GSM8K 65.5% [62.9, 68.0], MMLU-Pro 0-shot 48.1% [44.5, 51.8]. The two checkpoints differ by −0.71 pp HellaSwag on a paired 14,319-item test, so those figures are indicative here but were not measured on this checkpoint.

⚡ enable_thinking=False is 8.5× faster but costs 17–29 accuracy points (HumanEval −28.66, MMLU-Pro −21.43, GSM8K −17.36; all p<0.0001, paired McNemar). ZAYA1's accuracy is its reasoning, and its reasoning is what makes it slow — they cannot be separated. Use the flag for per-request routing (chat_template_kwargs), not as a global switch. If you need low latency more than accuracy, a weight-only quant of a non-reasoning model will serve you better. MMLU-Pro is 0-shot and is not comparable to Zyphra's 5-shot 74.2% — that gap is a protocol difference, not quantization damage. Full analysis: RESEARCH.md §5.22.

Budget-forced GPQA-Diamond using an s1-style harness that caps the reasoning trace and scores only the closed answer — stock lm-eval harnesses score near random on this model because ZAYA never closes its <think> block within a normal budget and answers in \boxed{} format:

think budget	GPQA-Diamond	traces self-closing </think>
2,500	45.8%	1/24
5,000	45.8%	2/24
12,000	62.5%	9/24

Sample size caveat. n=24 (paired). The 95% binomial confidence interval at 62.5% is roughly 41–81% — about 40 points wide. This is not evidence of parity with Zyphra's BF16 CoT figure of 71.0%; the interval is too wide to distinguish most hypotheses. What the data does support is the monotonic rise with reasoning budget, which is a checkpoint-health signal: a damaged checkpoint would not improve with more think tokens. A higher-n run is the most valuable open item on this checkpoint.

The gap to BF16 is consistent with the 16 GB context/reasoning-budget ceiling on the source hardware rather than quantization damage, but at n=24 that remains an interpretation, not a measurement.

License

Apache 2.0 — matches the ZAYA1-8B upstream license.