| --- |
| license: apache-2.0 |
| base_model: InternScience/Agents-A1-4B |
| base_model_relation: quantized |
| tags: |
| - nvfp4 |
| - compressed-tensors |
| - llm-compressor |
| - vllm |
| - qwen3_5 |
| - agent |
| --- |
| |
| # Agents-A1-4B-NVFP4 |
|
|
| NVFP4 (W4A4, compressed-tensors) quantization of [InternScience/Agents-A1-4B](https://huggingface.co/InternScience/Agents-A1-4B) — a 4B agentic model on the Qwen3.5 dense architecture (`Qwen3_5ForConditionalGeneration`: hybrid DeltaNet linear attention + full attention every 4 layers, Qwen3-VL vision tower, 262K context). |
|
|
| - **Size:** 4.21 GB (from ~8 GB bf16), incl. 0.24 GB bf16 MTP draft head |
| - **Bonus — grafted MTP:** the source checkpoint declares `mtp_num_hidden_layers: 1` but ships no `mtp.*` weights. This repo grafts the 15-tensor bf16 MTP draft head from the base model [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) (identical text-config dims), enabling speculative decoding: **63–66% draft acceptance, ~+20% single-stream decode** (measured, see below) |
| - **Scheme:** NVFP4 W4A4, group size 16, via llm-compressor 0.11.0 / compressed-tensors 0.16.0 |
| - **Kept bf16:** `lm_head`, vision tower (`model.visual*`), DeltaNet `conv1d` |
| - **Calibration:** 32 samples × 8192 seq, `neuralmagic/calibration` (pure-CPU calibration — no GPU used in the bake) |
|
|
| ## Serve with vLLM |
|
|
| ```bash |
| vllm serve sakamakismile/Agents-A1-4B-NVFP4 |
| ``` |
|
|
| Quantization is auto-detected — no `--quantization` flag needed. Requires a GPU with FP4 support (SM120 Blackwell) for the NVFP4 kernels. |
|
|
| With speculative decoding (grafted MTP draft, ~+20% single-stream): |
|
|
| ```bash |
| vllm serve sakamakismile/Agents-A1-4B-NVFP4 \ |
| --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' |
| ``` |
|
|
| ## Measured throughput — single GPU (vLLM 0.22.0, RTX PRO 2000 Blackwell SM120, KV fp8, 512-tok decode) |
|
|
| | Config | single-stream (c1) | aggregate 8-way (c8) | vs base | |
| |---|---|---|---| |
| | base (no MTP) | 73.6 t/s | 492.9 t/s | — | |
| | **MTP n=3 (grafted)** | **88.6 t/s** | 474.5 t/s | **+20.4% c1** / −3.7% c8 | |
|
|
| SpecDecoding metrics (vLLM): mean acceptance length ~2.9, per-position acceptance 0.83 / 0.66 / 0.48, avg draft acceptance **63–66%** — remarkable for a draft head grafted from the pre-fine-tune base model across InternScience's 3-stage agentic distillation. As usual, MTP is a latency win (single-stream / low concurrency); at saturation the extra draft forward costs slightly more than it saves. |
|
|
| Speculative decoding is lossless — outputs are identical to base decoding. |
|
|
| ## Recipe |
|
|
| Baked (pure-CPU, llm-compressor) with the same validated recipe as [Qwen3.6-27B-MTP-pi-tune-NVFP4](https://huggingface.co/sakamakismile/Qwen3.6-27B-MTP-pi-tune-NVFP4) and [ThinkingCap-Qwen3.6-27B-NVFP4](https://huggingface.co/sakamakismile/ThinkingCap-Qwen3.6-27B-NVFP4) (same `qwen3_5` architecture family). |
|
|
