---
base_model: Zyphra/ZAYA1-8B
license: apache-2.0
tags:
  - nvfp4
  - w4a4
  - compressed-tensors
  - quantization
  - vllm
  - blackwell
  - moe
pipeline_tag: text-generation
---

# ZAYA1-8B NVFP4 W4A4

**NVFP4 W4A4 inference for Zyphra's ZAYA1-8B on consumer Blackwell (RTX 5070 Ti, SM120)** —
4-bit weights *and* 4-bit activations running on native CUTLASS FP4 tensor-core kernels,
at **102.6 tok/s single-stream / 407.4 tok/s batch-8** in under 6 GB of VRAM.

One other public NVFP4 W4A4 ZAYA1 checkpoint exists — built with NVIDIA ModelOpt
and validated on a 96 GB workstation Blackwell card, with no published accuracy or
throughput numbers. This checkpoint is, as far as we know, the first built with
compressed-tensors on a 16 GB **consumer** card, with its own measured accuracy
and throughput. Full build/debug writeup, quantization scripts, and reproduction
steps: [github.com/t-timms/zaya1-godspeed](https://github.com/t-timms/zaya1-godspeed).

## Highlights

| Result | Detail |
|--------|--------|
| **102.6 tok/s** single / **407.4 tok/s** batch-8 | vLLM + CUDA graphs on RTX 5070 Ti (12.8× over eager mode) |
| **5.99 GB checkpoint** | 936 Linears in packed NVFP4 W4A4, 384 outlier-sensitive Linears kept BF16 (mixed precision) |
| **Checkpoint verified healthy** | Budget-forced GPQA-Diamond rises monotonically with reasoning budget — 45.8% → 62.5% at a 12k-token think budget, within CI of Zyphra's BF16 CoT 71.0% |
| **vLLM SM120 source build** | `TORCH_CUDA_ARCH_LIST=12.0` build enabling `cutlass_scaled_fp4_mm_sm120a` + FP4 group MoE GEMM — kernels that ship in vLLM source but not in wheels |

## Quantization details

| Field | Value |
|---|---|
| Base model | `Zyphra/ZAYA1-8B` |
| Method | compressed-tensors, `float-quantized` |
| Scheme | NVFP4, `w4a4` (4-bit weights, 4-bit activations) |
| Group size | 16 (tensor-group strategy) |
| Ignored / kept BF16 | `lm_head`, router, norm, qkv, cca modules, plus 12 MoE layers with `max_abs > 500` (384 Linears) |
| Calibration | 977 samples, ARC-mix calibration set, 936 Linears calibrated |
| Compressor | `NVFP4PackedCompressor` (`weight_packed_dtype=uint8`, `weight_scale_dtype=float8_e4m3fn`) |
| Built on | RTX 5070 Ti, 16 GB VRAM, CUDA 13.0 |

See `quantization_manifest.json` in this repo for the full machine-readable config,
including the exact outlier-layer list and mixed-precision threshold.

## Why this is hard

ZAYA1-8B is an 80-layer MoE (760M active / 8.4B total) with Zyphra's CCA
(compressed convolutional attention) — no stock quantization path works out of
the box. Notably: W4A4 requires calibrating activation scales (not just
weights), the compressed-tensors calibration path has a silent NaN-producing
trap if you calibrate through the fake-quant `nn.Linear.forward` wrapper, and
the NVFP4 global-scale convention (`2688 / max_abs`, divisor form, block
scales pre-multiplied) is undocumented — getting it wrong produces silent
pad-token collapse, not an error. Full root-cause writeup in `RESEARCH.md` on
the [GitHub repo](https://github.com/t-timms/zaya1-godspeed).

## Usage

This checkpoint requires vLLM built from source with SM120 NVFP4 CUTLASS
kernels (stock wheels don't include them) — see the
[reproduce steps](https://github.com/t-timms/zaya1-godspeed#reproduce) in the
GitHub repo. Inference must run in `bfloat16` (not fp16/fp32).

```bash
# vLLM built from source with: TORCH_CUDA_ARCH_LIST=12.0
vllm serve Ttimms/zaya1-8b-nvfp4-w4a4 --dtype bfloat16
```

## Evaluation

Budget-forced GPQA-Diamond (n=24, paired), using an s1-style harness that caps
the reasoning trace and scores only the closed answer — stock lm-eval
harnesses score near-random on this model because ZAYA never closes its
`<think>` block within a normal budget:

| think budget | GPQA-Diamond | traces self-closing `</think>` |
|---|---|---|
| 2,500 | 45.8% | 1/24 |
| 5,000 | 45.8% | 2/24 |
| 12,000 | **62.5%** | 9/24 |

The gap to Zyphra's BF16 CoT (71.0%) is the 16 GB context/reasoning-budget
ceiling on the source hardware — not quantization damage.

## License

Apache 2.0 — matches the ZAYA1-8B upstream license.