README.md

---
license: apache-2.0
base_model: Qwen/Qwen3.5-4B
base_model_relation: quantized
pipeline_tag: text-generation
tags:
- qwen3.5
- quantization
- int4
- w4a16
- compressed-tensors
- quantization-aware-distillation
- efficient-inference
---

<!-- TODO(release): fill the arXiv id. -->

<h1 align="center">Qwen3.5-4B-QAD-W4A16</h1>

<div align="center">

[![paper](https://img.shields.io/badge/arXiv-2607.04244-b31b1b.svg)](https://arxiv.org/abs/2607.04244)
[![code](https://img.shields.io/badge/GitHub-full%20pipeline-181717?logo=github)](https://github.com/nota-github/adaptfm-quant-dflash)
<br/>
[![target model](https://img.shields.io/badge/%F0%9F%A4%97%20Target-Qwen3.5--4B--QAD--W4A16-yellow)](https://huggingface.co/nota-ai/Qwen3.5-4B-QAD-W4A16)
[![drafter model](https://img.shields.io/badge/%F0%9F%A4%97%20Drafter-Qwen3.5--4B--DFlash--GPTQ--W4A16-yellow)](https://huggingface.co/nota-ai/Qwen3.5-4B-DFlash-GPTQ-W4A16)

</div>

## Introduction

<p align="center">
  <img src="figure/pipeline.png" alt="Pipeline: QAD INT4 target distillation, two-stage block-diffusion DFlash drafter training, and INT4-GPTQ + SWA drafter optimization."  width="800">
</p>

We reduce Qwen3.5-4B inference latency on a single A10G by quantizing both the target model and its speculative-decoding drafter to INT4, while maintaining quality across all three gates (MMLU-Pro, IFEval, GPQA-Diamond). This repository reproduces our 3rd-place Efficient Qwen Competition ([AdaptFM @ ICML 2026](https://adaptfm.gitlab.io/)) entry end to end. It combines three techniques:

- **Quantization-Aware Distillation (QAD):** trains the INT4 target model to follow the original BF16 model's distribution, recovering the accuracy that post-training quantization (PTQ) loses.
- **Two-Stage Training for Block-Diffusion Drafter:** pretrains the DFlash drafter with the BF16 target, then finetunes it on the QAD-applied INT4 target so it aligns with the quantized target distribution better than direct QAD-target training does.
- **Drafter Optimization:** applies INT4 GPTQ and Sliding-Window Attention (SWA) to reduce drafter overhead while maintaining acceptance length.

**Hardware:** Final serving and evaluation run on a single NVIDIA A10G 24 GB GPU. Retraining the QAD target and drafter requires an 80 GB-class multi-GPU machine, with at least 2 GPUs and preferably 8 GPUs. The gated prompt dataset is a one-time ~42 GB download.

## Quickstart (vLLM)

```bash
vllm serve nota-ai/Qwen3.5-4B-QAD-W4A16 \
    --quantization compressed-tensors \
    --trust-remote-code
```

- For speculative decoding with the companion W4A16 DFlash drafter, see the **[drafter card](https://huggingface.co/nota-ai/Qwen3.5-4B-DFlash-GPTQ-W4A16)**.

## Citation

```bibtex
@article{kim2026quantize,
  title   = {Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B},
  author  = {Jaeyeon Kim and Jewon Lee and Bo-Kyeong Kim},
  journal = {arXiv preprint arXiv:2607.04244},
  year    = {2026}
}
```


## License & provenance

Apache-2.0, derived from [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B)
(Apache-2.0) with INT4 initialization from
[cyankiwi/Qwen3.5-4B-AWQ-4bit](https://huggingface.co/cyankiwi/Qwen3.5-4B-AWQ-4bit).
Distillation data derives from NVIDIA's [Nemotron-Post-Training-Dataset-v2](https://huggingface.co/datasets/nvidia/Nemotron-Post-Training-Dataset-v2) (gated;
see its terms).