donghyunli/Meta-Llama-3-70B-KronQ-W2A16-g128

Meta-Llama-3-70B — KronQ W2A16 g128 (packed int2)

Paper: arXiv:2607.07964 · Code: GitHub

Meta-Llama-3-70B quantized to 2-bit weights / 16-bit activations, group size 128, with KronQ (Kronecker-factored Hessian quantization). Packed int2 (~22 GB); fused dequant + BiIP CUDA kernel with per-group scales.

Results (WikiText-2, seqlen 2048)

Perplexity: 7.646

Zero-shot accuracy:

PIQA	ARC-E	ARC-C	HellaSwag	WinoGrande	BoolQ	OBQA	Average
79.00	76.35	49.74	72.21	73.80	80.70	43.40	67.89

(lm-evaluation-harness 0-shot; acc_norm for PIQA/HellaSwag/ARC, acc for WinoGrande/BoolQ.)

Usage

python eval_pretrained.py meta-llama/Meta-Llama-3-70B donghyunli/Meta-Llama-3-70B-KronQ-W2A16-g128 --ppl --zs --distribute

(70B: --distribute spreads across GPUs.)

Recipe

Group-128 asymmetric W2, weight-only, --alpha 0.25, --act_order, BiIP, raw H_G.

License

Derivative of Meta-Llama-3-70B — llama3 license.