---
library_name: transformers
base_model: Qwen/Qwen3.5-397B-A17B
tags:
  - qwen3.5
  - moe
  - pruned
  - reap
  - nvfp4
  - fp4
license: apache-2.0
pipeline_tag: text-generation
---

# Qwen3.5-397B-A17B — REAP 28% Pruned, NVFP4

A personal experiment in aggressive MoE pruning. The goal: fit Qwen3.5-397B on **2× 96GB Blackwell GPUs** with usable KV cache (~90K tokens), without losing quality.

## What this is

28% of experts removed using [REAP](https://arxiv.org/abs/2501.02348) (Routing-Expert Activation Pruning) with a saliency × activation-count ordering, then quantized to NVFP4 using llm-compressor. Final size: **~164GB**.

Pruning is **heterogeneous** — each layer retains a different number of experts based on global importance ranking. Early layers (which carry more redundancy) are pruned more aggressively; late layers are barely touched. This maximizes quality retention for a given size budget.

## Benchmark results (non-thinking, lm-eval-harness)

| Benchmark | This model | Nvidia NVFP4 (full, ~240GB) |
|-----------|----------:|----------:|
| IFEval | **92.55** | 91.20 |
| MMLU Redux (generative) | 90.94 | **91.24** |
| GSM8K CoT (Llama) | 96.74 | **96.80** |

28% fewer experts, 30%+ smaller on disk, and benchmark scores within noise of the full model.

## Requirements

- **vLLM** ≥ 0.16.1 (nightly dev builds from the cu130 index work)
- **Transformers** ≥ 5.3

## vLLM patches required

This model uses variable expert counts per layer (not a fixed number), which stock vLLM doesn't support yet. Two files need patching:

1. **`qwen3_next.py`** — read per-layer expert count instead of assuming a single global value
2. **`qwen3_5.py`** — infer expert count from tensor shape during weight loading

## Details

- **Base model:** Qwen3.5-397B-A17B
- **Pruning:** REAP with 1,188 curated calibration samples, saliency × count global ordering
- **Quantization:** NVFP4 (FP4 E2M1, group size 16, duo scaling)
- **Target hardware:** 2× RTX PRO 6000