---
license: apache-2.0
tags:
- hlwq
- qwen3.5
- quantized
- kv-cache-compression
base_model: Jackrong/Qwen3.5-9B-Neo
pipeline_tag: text-generation
arxiv: '2603.29078'
---

> [!IMPORTANT]
> **Naming notice (2026-04-10).** The "HLWQ" technique used in this model is being rebranded to **HLWQ (Hadamard-Lloyd Weight Quantization)**. The change is only the name; the algorithm and the weights in this repository are unchanged.
>
> The rebrand resolves a name collision with an unrelated, earlier KV cache quantization method also named HLWQ ([Han et al., arXiv:2502.02617, 2025](https://arxiv.org/abs/2502.02617)). HLWQ addresses **weight** quantization with a **deterministic Walsh-Hadamard rotation** and Lloyd-Max scalar codebook; Han et al.'s HLWQ addresses **KV cache** quantization with a **random polar rotation**. The two methods are technically distinct.
>
> Existing loaders that load this repository by ID continue to work without changes. Future model uploads will use the HLWQ name.
>
> Reference paper for this technique: [arXiv:2603.29078](https://arxiv.org/abs/2603.29078) (v2 in preparation; v1 still uses the old name).

# 🧊 Qwen3.5-9B-Neo-HLWQ-Q5

**9B on any GPU** with HLWQ. Download: **6.3 GB**.

| Metric | Value |
|---|---|
| **VRAM** | 6.3 GB (peak 6.4 GB) |
| **Speed** | 42.8 tok/s |
| **Download** | 6.3 GB |
| **KV Cache Q3** | 5.3x, zero overhead |
| **Dequant** | 11s |
| **Layers** | 249 quantized |

## 📊 Charts

![Compression](compression.png)
![KV Speed](kv_speed.png)
![Family](family.png)

## 🏆 GPU Support

| GPU | VRAM | Fits? |
|---|---|---|
| **RTX 4060** | 8 GB | ✅ |
| **T4** | 16 GB | ✅ |
| **RTX 3060** | 12 GB | ✅ |
| **RTX 4090** | 24 GB | ✅ |

## 🔬 KV Cache Compression

| Method | tok/s | Compression |
|---|---|---|
| FP16 | 42.8 | 1.0x |
| HLWQ Q3 | 43.4 | 5.3x |
| HLWQ Q2 | 43.3 | 8.0x |

## 🚀 Quick Start

```bash
pip install polarquant[all]
polarquant chat Jackrong/Qwen3.5-9B-Neo
```

## 🔧 Technical Details

- **Architecture**: Qwen3.5-9B — 32 layers (hybrid attention+linear), 4 KV heads, head_dim=128
- **Weight quantization**: Hadamard rotation (128x128) + Lloyd-Max Q5 + torchao INT4
- **KV cache**: Hadamard rotation (128x128) + Lloyd-Max Q3 + real bit-packing
- **Streaming loader**: Per-module INT4 — fits any GPU ≥ 8 GB

## 📖 Citation

```bibtex
@article{polarquant2025,
  title={HLWQ: Hadamard-Rotated Lloyd-Max Quantization for LLM Compression},
  author={Vicentino, Caio},
  journal={arXiv preprint arXiv:2603.29078},
  year={2025}
}
```

📄 [Paper](https://arxiv.org/abs/2603.29078) · 💻 [GitHub](https://github.com/caiovicentino/polarengine-vllm) · 📦 [PyPI](https://pypi.org/project/polarquant/)


---

## 🚀 Quick Start

### Install
```bash
pip install git+https://github.com/caiovicentino/polarengine-vllm.git
```

### Load & Generate (1 line!)
```python
from polarengine_vllm import HLWQModel

model = HLWQModel.from_pretrained("caiovicentino1/Qwen3.5-9B-Neo-HLWQ-Q5")
print(model.generate("Hello, how are you?", max_new_tokens=100))
```

### With KV Cache Compression (5.3x more context)
```python
model = HLWQModel.from_pretrained("caiovicentino1/Qwen3.5-9B-Neo-HLWQ-Q5", kv_cache_nbits=3)
# KV cache now uses 5.3x less memory — fit longer conversations!
print(model.generate("Explain quantum computing in detail.", max_new_tokens=500))
```

### Benchmark
```bash
polarquant bench caiovicentino1/Qwen3.5-9B-Neo-HLWQ-Q5 --ppl --chart
```

### Gradio Demo
```bash
polarquant demo caiovicentino1/Qwen3.5-9B-Neo-HLWQ-Q5 --share
```

## 📦 Method: HLWQ

**Hadamard Rotation + Lloyd-Max Optimal Centroids**

Unlike GGUF (uniform quantization), HLWQ places quantization levels where weight density is highest — mathematically proven optimal for Gaussian-distributed neural network weights.

```
HLWQ Q5 (cos_sim > 0.996) > GGUF Q5_K_M (~0.99) at same size
```

## 🔗 Links

- 📄 [Paper — arXiv:2603.29078](https://arxiv.org/abs/2603.29078)
- 💻 [GitHub — HLWQ-Engine](https://github.com/caiovicentino/polarengine-vllm)
- 📦 [PyPI — `pip install polarquant`](https://pypi.org/project/polarquant/)