---
base_model:
  - Zyphra/ZAYA1-8B-legacy
  - Zyphra/ZAYA1-8B
base_model_relation: quantized
license: apache-2.0
tags:
  - nvfp4
  - w4a4
  - compressed-tensors
  - quantization
  - vllm
  - blackwell
  - moe
  - reasoning
pipeline_tag: text-generation
---

# ZAYA1-8B NVFP4 W4A4

**NVFP4 W4A4 inference for Zyphra's ZAYA1-8B on consumer Blackwell (RTX 5070 Ti, SM120)** —
4-bit weights *and* 4-bit activations running on native CUTLASS FP4 tensor-core kernels,
at **9.51 tok/s single-stream / ~74 tok/s batch-8** (`enforce_eager=True`) from a
9.46 GB mixed-precision checkpoint, built and served within a 16 GB VRAM budget.
An earlier 102.6/407.4 tok/s figure was measured under CUDA graphs, a path
confirmed 2026-08-14 to corrupt output on this card — see "Known issue" below,
retracted.

Activation quantization is the distinguishing feature here. Vendor-official 4-bit
releases — Google's Gemma 4 `qat-w4a16`, Zyphra's own `ZAYA1-8B-MXFP4-Experts` —
are **weights-only**. This checkpoint quantizes activations too, across 936 Linears.

## A smaller companion checkpoint exists

[`Ttimms/zaya1-8b-nvfp4-w4a4-uniform`](https://huggingface.co/Ttimms/zaya1-8b-nvfp4-w4a4-uniform)
is the same model with **zero BF16 exemptions** — every one of 1,320 Linears in
packed NVFP4 W4A4, at **6.02 GB** instead of 9.46 GB.

| | this checkpoint | `-uniform` |
|---|---:|---:|
| Size | 9.46 GB | **6.02 GB** (−36%) |
| BF16-exempted Linears | 384 | **0** |
| HellaSwag `acc` (n=10,042) | **46.49%** | 45.79% |
| HellaSwag `acc_norm` | **61.34%** | 60.65% |

The 384 exemptions cost **3.44 GB** and buy **0.71 pp** of HellaSwag accuracy
(95% CI [−1.26, −0.15], paired exact-binomial McNemar over 14,319 items). Take
this checkpoint for best measured accuracy; take `-uniform` for KV/context
headroom on a 16 GB card. Full paired table on the `-uniform` card.

This checkpoint is also the **control** for that evaluation, so it is kept
unchanged.

> **Manifest metadata note.** `quantization_manifest.json` in this repo records
> `"model": "Zyphra/ZAYA1-8B"`. The actual base is the pre-refactor 80-layer
> config, now published as `Zyphra/ZAYA1-8B-legacy` (see below). The manifest
> field predates Zyphra's June 2026 refactor and is left as-is rather than
> rewritten, because this artifact is a published evaluation control.

### ⚠️ Known issue: CUDA graph capture corrupts output on SM120

**The 102.6 / 407.4 tok/s figures previously published here were measured with
CUDA graphs enabled — a code path confirmed 2026-08-14 to produce numerically
wrong output on this card, independent of MoE backend.** A sweep of
`flashinfer_cutlass` (default), `cutlass`, and `marlin` (weight-only) all
produced garbage under graph capture; only `enforce_eager=True` generated
correctly. Since Marlin barely touches the FP4 MoE path and still failed, the
fault is graph capture itself, not any one kernel. Two adjacent upstream issues
exist — [CUTLASS #3096](https://github.com/NVIDIA/cutlass/issues/3096)
(different, non-graph-capture bug) and [FlashInfer #2776](https://github.com/flashinfer-ai/flashinfer/issues/2776)
(graph-capture-specific, but its stated root cause doesn't explain Marlin
failing too) — neither currently offers a fix that preserves CUDA graphs for
this failure mode. See [Pape, Evertz & Schönherr (arXiv:2605.19537)](https://arxiv.org/abs/2605.19537)
for the general phenomenon of backend-dependent correctness drift in LLM
serving. Full sweep, coherence re-verification, and citations in `RESEARCH.md`
and `ROADMAP.md` on the [GitHub repo](https://github.com/t-timms/zaya1-nvfp4-w4a4).

**Set `--enforce-eager` (or `enforce_eager=True`) when serving this checkpoint.**
It is the only configuration confirmed to produce coherent output.

### Throughput and memory (measured, `enforce_eager=True`)

Measured with vLLM's own benchmark CLI, not a bespoke harness. 5 separate
process invocations per configuration, GPU otherwise idle:

```bash
vllm bench latency --model <checkpoint> --dtype bfloat16 --kv-cache-dtype fp8 \
  --gpu-memory-utilization <frac> --max-model-len 4096 --no-enable-prefix-caching \
  --input-len 128 --output-len 256 --batch-size <1|8> --enforce-eager \
  --num-iters-warmup 1 --num-iters 1
```

| | 6.02 GB uniform | 9.46 GB mixed |
|---|---:|---:|
| Single-stream, median (range) | 9.52 (9.48–9.84) tok/s | **9.51** (9.45–9.81) tok/s |
| Batch-8, median (range) | 73.4 (72.2–74.9) tok/s | **74.4** (72.8–75.7) tok/s |
| Batch-8 scaling vs. batch-1 | 7.71× (96% of ideal) | 7.82× (98% of ideal) |
| `--gpu-memory-utilization` | 0.85 | **0.92** |

**Throughput is still unchanged between checkpoints** — the two land within
run-to-run noise of each other at both batch sizes. That part of the original
claim held; only the absolute numbers were wrong.

**Batching is near-ideal** (96–98% of theoretical linear scaling on batch-8),
which is evidence the MoE decode step's per-step cost does not grow meaningfully
with batch size.

**The two checkpoints cannot be run at the same memory fraction on a 16 GB
card.** This checkpoint fails at `--gpu-memory-utilization 0.85` with
`ValueError: No available memory for the cache blocks`, and at `1.0` with
`Free memory on device cuda:0 (14.66/15.92 GiB) ... less than desired` because a
desktop session holds ~1.3 GiB. Its working range is narrow; the `-uniform`
build runs at 0.85 with headroom — the practical case for the smaller
checkpoint, more so than the accuracy difference.

**Retracted, do not cite:** 102.6 tok/s single-stream / 407.4 tok/s batch-8
(CUDA graphs; coherent output was never verified at that speed).

> **An unverified external signal exists and disagrees with the table above.**
> llama.cpp PR #23112's own author reports 45.9 tok/s on a *slower* RTX 4070
> Ti (Q4_K_M GGUF) — beating this checkpoint's 9.5 tok/s by ~4.8×. Five
> attempts to reproduce it on this project's own SM120 hardware hit the same
> non-deterministic hang each time, ruling out the model and toolkit version
> as causes; looks like a WSL2/driver-level issue. Not resolved as of
> 2026-08-14 — see [GitHub repo](https://github.com/t-timms/zaya1-nvfp4-w4a4)
> `RESEARCH.md` §5.16 for the full diagnostic log.
>
> **Update, same day:** the gap likely has a principled cause independent of
> the hang above. Activation quantization (W4A4) gives no speed benefit at
> batch-1 — decode there is memory-bandwidth-bound, and quantizing
> activations only helps when compute is the bottleneck. Weight-only
> quantization is expected to win at batch-1 by design; this checkpoint's
> advantage is memory footprint and batched throughput (see the batch-8
> row above). Detail: `RESEARCH.md` §5.17.
>
> **A real batch-1 lever that does exist:** vLLM's built-in n-gram speculative
> decoding gives a validated **2.2× speedup on coding-edit prompts** (zero
> training, one config flag) — no gain on free-form generation, as expected.
> **Deployed to the production serve script and validated live** through the
> real OpenAI-compatible API. Detail: `RESEARCH.md` §5.18.

## Prior art and scope of claims

[`switzerchees/ZAYA1-8B-NVFP4`](https://huggingface.co/switzerchees/ZAYA1-8B-NVFP4)
(2026-05-19) is a genuine NVFP4 W4A4 ZAYA1-8B built with NVIDIA ModelOpt, and it
predates this checkpoint by two months. Its manifest reports
`gpu_capability: [12, 0]` — the same SM120 compute capability as an RTX 5070 Ti —
so **SM120 support is not a distinction of this work**, and this is not the first
W4A4 ZAYA1 checkpoint.

What is distinct, and all that is claimed:

| Dimension | switzerchees | This checkpoint |
|---|---|---|
| Toolchain | NVIDIA ModelOpt v0.44.0 | compressed-tensors / llm-compressor |
| Hardware | RTX PRO 6000, 96 GB workstation | RTX 5070 Ti, **16 GB consumer** |
| vLLM | Zyphra prebuilt `zaya1-pr` | hand-built SM120 CUTLASS from source |
| Accuracy published | none | budget-forced GPQA-Diamond (below), plus HumanEval / GSM8K / MMLU-Pro with confidence intervals |
| Throughput published | none | 9.5 tok/s single / ~74 tok/s batch-8, `enforce_eager`, median of 5 invocations. (An earlier 102.6 / 407.4 figure was measured under CUDA graphs and is retracted.) |
| Outlier handling | not addressed | mixed-precision exemption of 12 MoE layers |

## Base model revision — read before reproducing

This checkpoint was quantized from the **original 80-layer** ZAYA1-8B config
(`num_hidden_layers: 80`, `moe_router_topk`, `zaya_use_eda` / `zaya_use_mod`,
transformers 4.57.1).

In late June 2026 Zyphra refactored ZAYA1-8B into upstream-transformers form.
`Zyphra/ZAYA1-8B` now reports `num_hidden_layers: 40` with `layer_types: hybrid`
and `num_experts_per_tok`, and the original was moved to
[`Zyphra/ZAYA1-8B-legacy`](https://huggingface.co/Zyphra/ZAYA1-8B-legacy).

Core dimensions are unchanged (hidden 2048, 16 experts, top-1 routing, vocab
262272), so this reads as a re-expression of the same model rather than a new
one — but **reproduce against `Zyphra/ZAYA1-8B-legacy`**, or you will hit an
architecture mismatch.

**Verified, not just inferred (2026-08-14):** fetched `model.embed_tokens.weight`
directly from both repos via HTTP range request (no full download) and
compared the raw bytes — **byte-for-byte identical**, 1,074,266,112 bytes,
BF16, shape `[262272, 2048]`. The two repos' safetensors index files also
report an identical aggregate `total_size` (17,680,978,928 bytes) despite the
tensor layout changing from 2,483 named tensors (legacy) to 1,283 (current,
fused/batched expert tensors, transformers-conventional naming). This is why
`base_model` lists both repos below with `base_model_relation: quantized` —
not an assumption, a direct measurement on the one tensor that could be
checked without reverse-engineering the expert-fusion mapping. Every other
published NVFP4/GGUF/BNB quantization of this model tags only the current
`Zyphra/ZAYA1-8B` — this checkpoint is the only one built from and verified
against the pre-refactor structure directly.

## Highlights

| Result | Detail |
|--------|--------|
| **9.51 tok/s** single / **~74 tok/s** batch-8 | `enforce_eager=True` on RTX 5070 Ti — CUDA graphs produce corrupted output on this card, see "Known issue" above. Self-measured, 5 process invocations per config |
| **9.46 GB checkpoint** | 936 Linears in packed NVFP4 W4A4, 384 outlier-sensitive Linears kept BF16. The BF16 exemptions are why this is larger than a uniformly-quantized export — see [`-uniform`](https://huggingface.co/Ttimms/zaya1-8b-nvfp4-w4a4-uniform) (6.02 GB) for that build |
| **Checkpoint verified healthy** | Budget-forced GPQA-Diamond rises monotonically with reasoning budget, 45.8% → 62.5%. See the caveat on sample size below |
| **vLLM SM120 source build** | `TORCH_CUDA_ARCH_LIST=12.0` enabling `cutlass_scaled_fp4_mm_sm120a` + FP4 group MoE GEMM — kernels in vLLM source but not in wheels |

## Quantization details

| Field | Value |
|---|---|
| Base model | `Zyphra/ZAYA1-8B-legacy` (pre-refactor 80-layer config) |
| Method | compressed-tensors, `float-quantized` |
| Scheme | NVFP4, `w4a4` (4-bit weights, 4-bit activations) |
| Group size | 16 (tensor-group strategy) |
| Ignored / kept BF16 | `lm_head`, router, norm, qkv, cca modules, plus 12 MoE layers with `max_abs > 500` (384 Linears) |
| Calibration | 977 samples, ARC-mix calibration set, 936 Linears calibrated |
| Compressor | `NVFP4PackedCompressor` (`weight_packed_dtype=uint8`, `weight_scale_dtype=float8_e4m3fn`) |
| Built on | RTX 5070 Ti, 16 GB VRAM, CUDA 13.0 |

See `quantization_manifest.json` for the full machine-readable config, including
the exact outlier-layer list and mixed-precision threshold.

## Why this is hard

ZAYA1-8B is an 80-layer MoE (760M active / 8.4B total) with Zyphra's CCA
(compressed convolutional attention) — no stock quantization path works out of
the box. W4A4 requires calibrating activation scales, not just weights; the
compressed-tensors calibration path has a silent NaN-producing trap if you
calibrate through the fake-quant `nn.Linear.forward` wrapper; and the NVFP4
global-scale convention (`2688 / max_abs`, divisor form, block scales
pre-multiplied) is undocumented — getting it wrong produces silent pad-token
collapse, not an error. Full root-cause writeup in `RESEARCH.md` on the
[GitHub repo](https://github.com/t-timms/zaya1-nvfp4-w4a4).

## Usage

Requires vLLM built from source with SM120 NVFP4 CUTLASS kernels (stock wheels
don't include them) — see the
[reproduce steps](https://github.com/t-timms/zaya1-nvfp4-w4a4#reproduce).
Inference must run in `bfloat16` (not fp16/fp32).

```bash
# vLLM built from source with: TORCH_CUDA_ARCH_LIST=12.0
vllm serve Ttimms/zaya1-8b-nvfp4-w4a4 --dtype bfloat16 --enforce-eager \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 5, "prompt_lookup_min": 2}'
```

**`--enforce-eager` is required for correct output**, not just recommended —
CUDA graph capture corrupts generation on this card regardless of MoE backend.
See "Known issue" above before serving without it.

**`--speculative-config` is optional but free** — lossless n-gram speculative
decoding, validated **2.2× faster on coding-edit prompts** (no gain on
free-form generation, since there's no prompt/output overlap to exploit).
Detail: `RESEARCH.md` §5.18. The exact command above is
[`scripts/serve.sh`](https://github.com/t-timms/zaya1-nvfp4-w4a4/blob/master/scripts/serve.sh)
in the repo.

## Evaluation

> **Generative benchmarks were measured on the companion 6.02 GB uniform
> checkpoint** ([`Ttimms/zaya1-8b-nvfp4-w4a4-uniform`](https://huggingface.co/Ttimms/zaya1-8b-nvfp4-w4a4-uniform)),
> not this one: HumanEval **72.6%** pass@1 (95% CI [65.3, 78.8]), GSM8K
> **65.5%** [62.9, 68.0], MMLU-Pro 0-shot **48.1%** [44.5, 51.8]. The two
> checkpoints differ by −0.71 pp HellaSwag on a paired 14,319-item test, so
> those figures are indicative here but were not measured on this checkpoint.
>
> ⚡ **`enable_thinking=False` is 8.5× faster but costs 17–29 accuracy points**
> (HumanEval −28.66, MMLU-Pro −21.43, GSM8K −17.36; all p<0.0001, paired
> McNemar). **ZAYA1's accuracy *is* its reasoning, and its reasoning *is* what
> makes it slow — they cannot be separated.** Use the flag for *per-request
> routing* (`chat_template_kwargs`), not as a global switch. If you need low
> latency more than accuracy, a weight-only quant of a non-reasoning model
> will serve you better.
> **MMLU-Pro is 0-shot and is not comparable to Zyphra's 5-shot 74.2%** — that
> gap is a protocol difference, not quantization damage. Full analysis:
> [`RESEARCH.md` §5.22](https://github.com/t-timms/zaya1-nvfp4-w4a4/blob/master/RESEARCH.md).

Budget-forced GPQA-Diamond using an s1-style harness that caps the reasoning
trace and scores only the closed answer — stock lm-eval harnesses score near
random on this model because ZAYA never closes its `<think>` block within a
normal budget and answers in `\boxed{}` format:

| think budget | GPQA-Diamond | traces self-closing `</think>` |
|---|---|---|
| 2,500 | 45.8% | 1/24 |
| 5,000 | 45.8% | 2/24 |
| 12,000 | **62.5%** | 9/24 |

> **Sample size caveat.** n=24 (paired). The 95% binomial confidence interval at
> 62.5% is roughly 41–81% — about 40 points wide. This is **not** evidence of
> parity with Zyphra's BF16 CoT figure of 71.0%; the interval is too wide to
> distinguish most hypotheses. What the data does support is the *monotonic*
> rise with reasoning budget, which is a checkpoint-health signal: a damaged
> checkpoint would not improve with more think tokens. A higher-n run is the
> most valuable open item on this checkpoint.

The gap to BF16 is consistent with the 16 GB context/reasoning-budget ceiling on
the source hardware rather than quantization damage, but at n=24 that remains an
interpretation, not a measurement.

## License

Apache 2.0 — matches the ZAYA1-8B upstream license.