donghyunli/Meta-Llama-3-8B-KronQ-W4A16

Meta-Llama-3-8B — KronQ W4A16 (packed int4)

Paper: arXiv:2607.07964 · Code: GitHub

Meta-Llama-3-8B quantized to 4-bit weights / 16-bit activations with KronQ (Kronecker-factored Hessian quantization). Weights are stored packed int4 (~6.3 GB vs 16 GB fp16; the 128k-token embedding + lm_head stay fp16); a fused dequant + bidirectional-incoherence (BiIP) CUDA kernel unpacks them on the fly at inference.

Results (WikiText-2, seqlen 2048)

Perplexity: 6.42

Zero-shot accuracy:

PIQA	ARC-E	ARC-C	HellaSwag	WinoGrande	BoolQ	OBQA	Average
79.43	78.16	51.45	78.24	73.16	82.23	44.20	69.55

(lm-evaluation-harness, 0-shot. acc_norm for PIQA/HellaSwag/ARC/OBQA, acc for WinoGrande/BoolQ.)

Usage

KronQ-packed checkpoint (model.safetensors carries biip_w_codes/scale/zero + BiIP buffers, see kronq_packed_config.json). Load with the KronQ runtime:

# clone https://github.com/<...>/KronQ and build the CUDA kernels, then:
python eval_pretrained.py meta-llama/Meta-Llama-3-8B donghyunli/Meta-Llama-3-8B-KronQ-W4A16 --ppl --zs

Recipe

Per-channel asymmetric W4, weight-only (a_bits=16), --alpha 0.25, bidirectional incoherence processing (BiIP, Hadamard kernel), act_order. Calibrated on 128 WikiText-2 sequences with raw H_G. To reproduce instead of using these weights, see the KronQ repo's "Reproduce" path.

License

Derivative of Meta-Llama-3-8B — subject to the Llama 3 Community License.