---
base_model: meta-llama/Llama-2-7b-hf
language:
- en
license: llama2
pipeline_tag: text-generation
tags:
- kronq
- quantization
- int4
- weight-only
---

# Llama-2-7b — KronQ W4A16 (packed int4)

**Paper:** [arXiv:2607.07964](https://arxiv.org/abs/2607.07964) · **Code:** [GitHub](https://github.com/Intelligent-Computing-Lab-Panda/KronQ)

[Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b-hf) quantized to **4-bit weights / 16-bit activations** with **KronQ** (Kronecker-factored Hessian quantization). Weights are stored **packed int4** (~3.8 GB vs 13 GB fp16); a fused dequant + bidirectional-incoherence (BiIP) CUDA kernel unpacks them on the fly at inference.

## Results (WikiText-2, seqlen 2048)

**Perplexity:** **5.56**

**Zero-shot accuracy:**

| PIQA | ARC-E | ARC-C | HellaSwag | WinoGrande | BoolQ | OBQA | Average |
|---|---|---|---|---|---|---|---|
| 78.67 | 72.39 | 44.88 | 75.17 | 68.43 | 77.37 | 44.40 | **65.90** |

(lm-evaluation-harness, 0-shot. `acc_norm` for PIQA/HellaSwag/ARC/OBQA, `acc` for WinoGrande/BoolQ.)

## Usage

This is a **KronQ-packed** checkpoint (custom format: `model.safetensors` carries `biip_w_codes`/scale/zero + BiIP buffers, described in `kronq_packed_config.json`). Load it with the KronQ runtime:

```bash
# clone https://github.com/Intelligent-Computing-Lab-Panda/KronQ and build the CUDA kernels, then:
python eval_pretrained.py meta-llama/Llama-2-7b-hf donghyunli/Llama-2-7b-KronQ-W4A16 --ppl --zs
```

## Recipe

Per-channel asymmetric W4, weight-only (a_bits=16), `--alpha 0.5`, bidirectional incoherence processing (BiIP, Hadamard kernel), `act_order`. Calibrated on 128 WikiText-2 sequences. To reproduce the quantization yourself (instead of using these weights), see the KronQ repo's "Reproduce" path + the companion Hessian repo.

## License

Derivative of Llama-2-7b — subject to the [Llama 2 Community License](https://ai.meta.com/llama/license/).