---
base_model:
  - Zyphra/ZAYA1-8B-legacy
  - Zyphra/ZAYA1-8B
base_model_relation: quantized
license: apache-2.0
tags:
  - nvfp4
  - w4a4
  - compressed-tensors
  - quantization
  - vllm
  - blackwell
  - moe
  - reasoning
pipeline_tag: text-generation
---

# ZAYA1-8B NVFP4 W4A4 — uniform (no BF16 exemptions)

**6.02 GB.** Every one of 1,320 Linears in packed NVFP4 W4A4 — 4-bit weights *and*
4-bit activations — with **zero BF16 exemptions**, served on a 16 GB consumer
Blackwell card (RTX 5070 Ti, SM120) via native CUTLASS FP4 tensor-core kernels.

This is the companion to
[`Ttimms/zaya1-8b-nvfp4-w4a4`](https://huggingface.co/Ttimms/zaya1-8b-nvfp4-w4a4)
(9.46 GB), which keeps 384 outlier-sensitive Linears at BF16. **Both are
maintained. Neither supersedes the other** — the choice is 3.44 GB of VRAM
against a small, measured accuracy cost.

## Which one should you use?

| | this checkpoint | [9.46 GB variant](https://huggingface.co/Ttimms/zaya1-8b-nvfp4-w4a4) |
|---|---:|---:|
| Weights (`model.safetensors`) | **5.99 GB** | 9.46 GB |
| Repository total | **6.02 GB** | 9.49 GB |
| BF16-exempted Linears | **0** | 384 |
| W4A4 Linears | 1,320 | 936 |
| HellaSwag `acc` (n=10,042) | 45.79% | **46.49%** |
| HellaSwag `acc_norm` | 60.65% | **61.34%** |
| KV cache headroom on 16 GB | **6.83 GiB / ~336k tokens** | materially less |

**Take this one** if you want maximum context/KV headroom on a 16 GB card, or you
are batching. The accuracy cost is −0.71 pp on HellaSwag.

**Take the 9.46 GB one** if you want the best measured accuracy and can afford the
VRAM, or if you need the exact artifact used as the control in the evaluation below.

> **On the two size figures.** `6.02 GB` is the uniform build's **whole repository**;
> `9.46 GB` is the mixed build's **`model.safetensors` alone** (its repository total
> is 9.49 GB). Both figures are individually correct and both are already published,
> but they measure different things — so the table above gives each basis
> explicitly. The reduction holds either way: **−36.7%** on weights, **−36.6%** on
> repository totals. Measured from the Hub, 2026-08-09.


### ⚠️ Known issue: CUDA graph capture corrupts output on SM120

**Throughput figures previously published on this card (104.7 tok/s here /
407.4 tok/s batch-8 elsewhere in this project) were measured with CUDA graphs
enabled — a code path confirmed 2026-08-14 to produce numerically wrong output
on this hardware, independent of MoE backend.** A sweep of `flashinfer_cutlass`
(default), `cutlass`, and `marlin` (weight-only) all produced garbage under
graph capture; only `enforce_eager=True` generated correctly. Since Marlin
barely touches the FP4 MoE path and still failed, the fault is graph capture
itself, not any one kernel. Two adjacent upstream issues exist —
[CUTLASS #3096](https://github.com/NVIDIA/cutlass/issues/3096) (different,
non-graph-capture bug) and [FlashInfer #2776](https://github.com/flashinfer-ai/flashinfer/issues/2776)
(graph-capture-specific, but its stated root cause doesn't explain Marlin
failing too) — neither currently offers a fix that preserves CUDA graphs for
this failure mode. See [Pape, Evertz & Schönherr (arXiv:2605.19537)](https://arxiv.org/abs/2605.19537)
for the general phenomenon of backend-dependent correctness drift in LLM
serving. Full sweep, coherence re-verification, and citations in `RESEARCH.md`
and `ROADMAP.md` on the [GitHub repo](https://github.com/t-timms/zaya1-nvfp4-w4a4).

**Set `--enforce-eager` (or `enforce_eager=True`) when serving this checkpoint.**
It is the only configuration confirmed to produce coherent output.

### Throughput and memory (measured, `enforce_eager=True`)

Measured with vLLM's own benchmark CLI, not a bespoke harness. 5 separate
process invocations per configuration, GPU otherwise idle:

```bash
vllm bench latency --model <checkpoint> --dtype bfloat16 --kv-cache-dtype fp8 \
  --gpu-memory-utilization <frac> --max-model-len 4096 --no-enable-prefix-caching \
  --input-len 128 --output-len 256 --batch-size <1|8> --enforce-eager \
  --num-iters-warmup 1 --num-iters 1
```

| | 6.02 GB uniform | 9.46 GB mixed |
|---|---:|---:|
| Single-stream, median (range) | **9.52** (9.48–9.84) tok/s | 9.51 (9.45–9.81) tok/s |
| Batch-8, median (range) | **73.4** (72.2–74.9) tok/s | 74.4 (72.8–75.7) tok/s |
| Batch-8 scaling vs. batch-1 | 7.71× (96% of ideal) | 7.82× (98% of ideal) |
| `--gpu-memory-utilization` | **0.85** | 0.92 |

**Throughput is still unchanged between checkpoints** — the two land within
run-to-run noise of each other at both batch sizes. That part of the original
claim held; only the absolute numbers were wrong.

**Batching is near-ideal** (96–98% of theoretical linear scaling on batch-8),
which is evidence the MoE decode step's per-step cost does not grow meaningfully
with batch size.

**The two checkpoints cannot be run at the same memory fraction on a 16 GB
card.** The mixed checkpoint fails at `--gpu-memory-utilization 0.85` with
`ValueError: No available memory for the cache blocks`, and at `1.0` with
`Free memory on device cuda:0 (14.66/15.92 GiB) ... less than desired` because a
desktop session holds ~1.3 GiB. Its working range is narrow; the uniform build
runs at 0.85 with headroom — the practical case for the smaller checkpoint,
more so than the accuracy difference.

**Retracted, do not cite:** 104.7 / 105.3 tok/s single-stream, 102.6 / 407.4
tok/s from elsewhere in this project (all measured under CUDA graphs).

> **An unverified external signal exists and disagrees with the table above.**
> llama.cpp PR #23112's own author reports 45.9 tok/s on a *slower* RTX 4070
> Ti (Q4_K_M GGUF) — beating this checkpoint's 9.5 tok/s by ~4.8×. Five
> attempts to reproduce it on this project's own SM120 hardware hit the same
> non-deterministic hang each time, ruling out the model and toolkit version
> as causes; looks like a WSL2/driver-level issue. Not resolved as of
> 2026-08-14 — see [GitHub repo](https://github.com/t-timms/zaya1-nvfp4-w4a4)
> `RESEARCH.md` §5.16 for the full diagnostic log.
>
> **Update, same day:** the gap likely has a principled cause independent of
> the hang above. Activation quantization (W4A4) gives no speed benefit at
> batch-1 — decode there is memory-bandwidth-bound, and quantizing
> activations only helps when compute is the bottleneck. Weight-only
> quantization is expected to win at batch-1 by design; this checkpoint's
> advantage is memory footprint and batched throughput (see the batch-8
> row above). Detail: `RESEARCH.md` §5.17.
>
> **A real batch-1 lever that does exist:** vLLM's built-in n-gram speculative
> decoding gives a validated **2.2× speedup on coding-edit prompts** (zero
> training, one config flag) — no gain on free-form generation, as expected.
> **Deployed to the production serve script and validated live** through the
> real OpenAI-compatible API. Detail: `RESEARCH.md` §5.18.

## Accuracy: measured, paired, and honest about its limits

The two checkpoints were compared with a **paired exact-binomial McNemar test on
discordant items**, joined per `doc_id`, over **14,319 items each** across four
pure-loglikelihood tasks. No generation is involved, so this is immune to the
`<think>`-never-terminates artifact that invalidates naive generative evals on
this model. No chat template — these are ranked-continuation tasks.

| task | metric | n | this (6.02 GB) | control (9.46 GB) | Δ pp | 95% CI | p |
|---|---|---:|---:|---:|---:|---|---:|
| hellaswag | acc | 10,042 | 45.79% | 46.49% | **−0.71** | [−1.26, −0.15] | 0.0140 |
| hellaswag | acc_norm | 10,042 | 60.65% | 61.34% | −0.70 | [−1.39, −0.01] | 0.0504 |
| arc_challenge | acc | 1,172 | 37.97% | 36.95% | +1.02 | [−1.42, +3.47] | 0.4522 |
| arc_challenge | acc_norm | 1,172 | 37.97% | 40.36% | −2.39 | [−4.97, +0.19] | 0.0799 |
| winogrande | acc | 1,267 | 56.20% | 59.04% | −2.84 | [−6.04, +0.36] | 0.0906 |
| piqa | acc | 1,838 | 69.42% | 70.02% | −0.60 | [−2.41, +1.21] | 0.5564 |
| piqa | acc_norm | 1,838 | 70.89% | 70.08% | +0.82 | [−1.02, +2.65] | 0.4166 |

> **Read the intervals, not the p-values.** No comparison survives Bonferroni
> correction (α = 0.05/7 = 0.0071). That is **absence of resolution, not evidence
> of absence.** HellaSwag is the only adequately powered task here, and its 95% CI
> **excludes zero**. The three smaller benchmarks cannot rule out regressions of
> **−4.97 pp** (arc_challenge `acc_norm`) or **−6.04 pp** (winogrande). Five of
> seven comparisons point negative, consistent with a small real regression that
> only HellaSwag has the samples to detect.
>
> **Defensible claim: −0.71 pp on HellaSwag for −36% checkpoint size.** Anything
> stronger is unsupported by this data.

An earlier ARC-Easy comparison suggested no cost at all. That was an **unpaired**
two-proportion test applied to paired data: both checkpoints are quantizations of
one base model scored on the same items, and discarding that correlation discards
the statistical power. It also happened to favour the wrong checkpoint. Aggregate
accuracy output cannot be converted into a paired test after the fact — it
requires re-running the model with per-item logging.

Reproduce with `scripts/run_phase_a.py` and `scripts/analyze_phase_a.py` in the
[GitHub repo](https://github.com/t-timms/zaya1-nvfp4-w4a4).

### Generative benchmarks (2026-08-16)

The tests above are **loglikelihood** — they score pre-written continuations,
so they measure *ranking*, never *producing*. These three measure generation:

| benchmark | score | 95% CI | n |
|---|---:|---|---:|
| **HumanEval** | **72.6%** pass@1 | [65.3, 78.8] | 164 |
| **GSM8K** | **65.5%** | [62.9, 68.0] | 1,319 |
| **MMLU-Pro** (0-shot) | **48.1%** | [44.5, 51.8] | 700 |

`enforce_eager=True`, think_budget 4096, temp 0.6 / top_p 0.95 (Zyphra's
published recommendation), seed 42. Reproduce with
`scripts/run_budget_forced_suite.sh`.

**On HumanEval:** published figures put Qwen 3 7B at roughly 68–72% and
Llama 3 8B at 62–65% — both at full precision. This checkpoint matches or
exceeds them at 4-bit weights *and* activations in 6.02 GB. Zyphra publishes
no HumanEval number for ZAYA1, so there is no retention figure to quote; this
appears to be the first measurement of it at any precision.

**On MMLU-Pro — read this before comparing to Zyphra's 74.2%.** That number is
not a valid comparison point for this one. lm-eval's MMLU-Pro task is
**5-shot**; this harness is **0-shot**, because ZAYA1's `<think>` trace
frequently never terminates and the standard harness scores the unterminated
trace as the answer (the artifact that put IFEval at 19.8% against an 85.58%
reference). Zyphra's figure also comes from a private harness with undisclosed
generation limits. The gap is a protocol difference, not quantization damage:
published INT4 loss on MMLU-Pro is ~1.6 pp, and the paired test above bounds
this checkpoint's cost at −0.71 pp HellaSwag.

**The budget was tested, not assumed.** Both benchmarks were re-run at
think_budget 8192 and compared with paired McNemar on identical items — GSM8K
**+0.15 pp** (p=0.9581), MMLU-Pro **+3.29 pp** (p=0.0673). Neither is
significant, so 4096 is the correct operating budget and more reasoning room
does not help this model. Detail: `RESEARCH.md` §5.22.

**On GSM8K:** honest but unremarkable — Llama 3.2 3B reports 77.7% at 8-shot.
This run is 0-shot, 78% of traces truncate, and GSM8K is heavily contaminated.
Treat it as a regression check that quantization did not break arithmetic
reasoning, not as evidence of mathematical ability.

### ⚡ `enable_thinking=False` — 8.5× faster, but it costs 17–29 accuracy points

ZAYA1's chat template (Zyphra's, not ours) supports disabling the reasoning
trace, which pre-closes `<think>` so the model answers immediately. **This is
the single biggest latency lever available, and it is not free.** Measured on
this checkpoint, paired McNemar on identical items:

| benchmark | thinking | `enable_thinking=False` | Δ | p | wall time |
|---|---:|---:|---:|---:|---|
| HumanEval | 72.6% | 43.9% | **−28.66 pp** | <0.0001 | 15 m → **2 m** |
| MMLU-Pro | 48.1% | 26.7% | **−21.43 pp** | <0.0001 | 39 m → **4 m** |
| GSM8K | 65.5% | 48.1% | **−17.36 pp** | <0.0001 | 63 m → **8 m** |

**Read this before deciding how to serve the model: ZAYA1's accuracy *is* its
reasoning, and its reasoning *is* what makes it slow — the two cannot be
separated.** It is not a fast interactive model, and no serving configuration
makes it one. If you need low latency more than accuracy, a weight-only quant
of a non-reasoning model will serve you better.

**Where the flag is genuinely useful: per-request routing, not a global
switch.** vLLM accepts it per request:

```json
{"model": "...", "messages": [...],
 "chat_template_kwargs": {"enable_thinking": false}}
```

Send mechanical work (file reads, trivial edits, simple tool calls) down the
fast path and keep reasoning for real problem-solving. Note that **12 / 53 /
173 items respectively were solved *only* with thinking disabled** — some tasks
are actively hurt by overthinking, so routing can win twice.

Full analysis: [`RESEARCH.md` §5.23](https://github.com/t-timms/zaya1-nvfp4-w4a4/blob/master/RESEARCH.md).

## Why the exemptions were removable

The 9.46 GB checkpoint exempts 12 MoE layers whose calibrated activation
`max_abs` exceeds 500 — the worst being 8,896 at `L75.experts.1.linear_fc2`,
622× the median. Only **24** `linear_fc2` modules actually offend, but
**FusedMoE requires uniform quantization per layer**, so protecting them forces
exemption of `fc1` *and* `fc2` across all 16 experts in each affected layer:
**384 Linears, 3.44 GB.** A 16× overpay, where 16 is `num_experts`.

The exemptions turned out to be largely redundant. SOAR — which targets the same
FP8 block-scale rounding error the outliers cause — was introduced *after* the
mixed-precision decision, and the combination was never re-tested. The worst
channel in the model quantizes coherently at W4A4 without them.

**No residual correction (ARCQuant or otherwise) is applied to this checkpoint,
and none is required.** The numbers above are for the artifact exactly as
published.

## Base model revision — read before reproducing

Quantized from the **original 80-layer** ZAYA1-8B config
(`num_hidden_layers: 80`, `moe_router_topk`, `zaya_use_eda` / `zaya_use_mod`,
transformers 4.57.1).

In late June 2026 Zyphra refactored ZAYA1-8B into upstream-transformers form:
`Zyphra/ZAYA1-8B` now reports `num_hidden_layers: 40` with `layer_types: hybrid`
and `num_experts_per_tok`, and the original moved to
[`Zyphra/ZAYA1-8B-legacy`](https://huggingface.co/Zyphra/ZAYA1-8B-legacy).

Core dimensions are unchanged (hidden 2048, 16 experts, top-1 routing, vocab
262272), so this is a re-expression rather than a new model — but **reproduce
against `Zyphra/ZAYA1-8B-legacy`**, or you will hit an architecture mismatch.

**Verified, not just inferred (2026-08-14):** fetched `model.embed_tokens.weight`
directly from both repos via HTTP range request (no full download) and
compared the raw bytes — **byte-for-byte identical**, 1,074,266,112 bytes,
BF16, shape `[262272, 2048]`. The two repos' safetensors index files also
report an identical aggregate `total_size` (17,680,978,928 bytes) despite the
tensor layout changing from 2,483 named tensors (legacy) to 1,283 (current,
fused/batched expert tensors, transformers-conventional naming). This is why
`base_model` lists both repos below with `base_model_relation: quantized` —
not an assumption, a direct measurement on the one tensor that could be
checked without reverse-engineering the expert-fusion mapping. Every other
published NVFP4/GGUF/BNB quantization of this model tags only the current
`Zyphra/ZAYA1-8B` — this checkpoint is the only one built from and verified
against the pre-refactor structure directly.

## Quantization details

| Field | Value |
|---|---|
| Base model | `Zyphra/ZAYA1-8B-legacy` (pre-refactor 80-layer config) |
| Method | compressed-tensors, `float-quantized` |
| Scheme | NVFP4, `w4a4` (4-bit weights, 4-bit activations) |
| Group size | 16 (tensor-group strategy) |
| W4A4 Linears | **1,320** |
| BF16-exempted | **0** |
| Ignored / kept BF16 | `lm_head`, router, norm, qkv, cca modules only |
| Calibration | 977 samples, ARC-mix set, 1,320 Linears calibrated, SOAR enabled |
| Compressor | `NVFP4PackedCompressor` (`weight_packed_dtype=uint8`, `weight_scale_dtype=float8_e4m3fn`) |
| Built on | RTX 5070 Ti, 16 GB VRAM, CUDA 13.0 |

`quantization_manifest.json` carries the full machine-readable config, the exact
outlier-layer list, and the evaluation block reproduced above.

## Usage

Requires vLLM **built from source** with SM120 NVFP4 CUTLASS kernels — stock
wheels do not compile them. Inference must run in `bfloat16`.

```bash
# vLLM built from source with: TORCH_CUDA_ARCH_LIST=12.0
vllm serve Ttimms/zaya1-8b-nvfp4-w4a4-uniform --dtype bfloat16 --enforce-eager \
  --speculative-config '{"method": "ngram", "num_speculative_tokens": 5, "prompt_lookup_max": 5, "prompt_lookup_min": 2}'
```

The exact command above is
[`scripts/serve.sh`](https://github.com/t-timms/zaya1-nvfp4-w4a4/blob/master/scripts/serve.sh)
in the repo.

Serving notes for this model family:

- `enable_prefix_caching=False` is **required** — ZAYA's CCA state is not
  cacheable, and vLLM defaults this to `True`.
- **`--enforce-eager` is required for correct output**, not just recommended —
  CUDA graph capture corrupts generation on this card regardless of MoE
  backend. See "Known issue" above before serving without it.
- **`--speculative-config` is optional but free** — lossless n-gram
  speculative decoding, validated **2.2× faster on coding-edit prompts** (no
  gain on free-form generation, since there's no prompt/output overlap to
  exploit). Detail: `RESEARCH.md` §5.18.
- Set `VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0` on a 16 GB card; the estimator
  otherwise reserves 3.5+ GiB and pushes the KV cache into OOM.
- Generative evals need budget forcing. ZAYA is a reasoning model whose `<think>`
  trace frequently never terminates within a normal token budget, and it answers
  in `\boxed{}` form — stock harnesses score it near random for reasons unrelated
  to quantization.

## Why this is hard

ZAYA1-8B is an 80-layer MoE (760M active / 8.4B total) with Zyphra's CCA
(compressed convolutional attention). W4A4 requires calibrating activation
scales, not just weights; the compressed-tensors calibration path has a silent
NaN-producing trap if you calibrate through the fake-quant `nn.Linear.forward`
wrapper; and the NVFP4 global-scale convention (`2688 / max_abs`, divisor form,
block scales pre-multiplied) is undocumented — getting it wrong produces silent
pad-token collapse rather than an error.

## Known limits

- Single-stream and batch-8 throughput are now both measured for both
  checkpoints with `vllm bench latency` under `enforce_eager=True` (see above):
  **9.52 tok/s single-stream / 73.4 tok/s batch-8 here**, essentially identical
  to the 9.46 GB build. The earlier CUDA-graph figures (104.7/105.3 single,
  102.6/407.4 elsewhere) are retracted — see "Known issue" above.
- No BF16 control eval exists — a BF16 ZAYA1-8B needs ~35 GiB and does not fit
  the source hardware. All accuracy claims here are *relative to the 9.46 GB
  checkpoint*, not to full precision.
- The three smaller benchmarks are underpowered, as stated above.

## License

Apache 2.0 — matches the ZAYA1-8B upstream license.