SmolLM2-360M-Instruct-GLQ-block-diagonal-4bpw

GLQ (E8-lattice) 4 bpw quantization of HuggingFaceTB/SmolLM2-360M-Instruct, using a block-diagonal randomized Hadamard transform — no power-of-2 FHT padding, so the stored size matches the nominal rate (a genuine 4 bpw, ~253 MB).

vs the earlier xv0y5ncu/SmolLM2-360M-Instruct-GLQ-4bpw: that checkpoint zero-padded non-power-of-2 dimensions up to the next power of two for the FHT, which inflated the effective rate to 6.4 bpw (348 MB). This block-diagonal version stores only the real weights — smaller, with an accurate bpw label. At equal storage, block-diagonal also yields higher SQNR, so it is a strict per-bit improvement.

Quality — lm-evaluation-harness 5-task

GLQ and bf16 measured on the same stack (vLLM backend, transformers 5.x).

Task	GLQ 4 bpw	bf16	GLQ / bf16
arc_easy	0.4941	0.4895	100.9%
hellaswag (acc_norm)	0.5386	0.5678	94.9%
piqa	0.6953	0.7100	97.9%
winogrande	0.5770	0.5746	100.4%
lambada_openai	0.4923	0.5162	95.4%
average	0.5595	0.5716	97.9%

Tasks: arc_easy, hellaswag, piqa, winogrande, lambada_openai (acc_norm where available, else acc). Per-task ratios slightly above 100% are within lm-eval noise.

Details

• Bits per weight: 4 bpw (uniform) — block-diagonal, so effective is approximately nominal.
• Checkpoint size: ~253 MB (bf16 original ~720 MB).
• Average SQNR: 18.6 dB.
• Calibration: WikiText-2, 128 samples, sequence length 2048.
• Method: E8 lattice codebook + randomized Hadamard transform (block-diagonal) + LDLQ.

Usage

pip install glq

import glq.hf_integration  # registers GLQ with transformers
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "xv0y5ncu/SmolLM2-360M-Instruct-GLQ-block-diagonal-4bpw"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, device_map="cuda", dtype=torch.float16)
ids = tok("The capital of France is", return_tensors="pt").to("cuda")
print(tok.decode(model.generate(**ids, max_new_tokens=20)[0], skip_special_tokens=True))

Serve with vLLM: vllm serve xv0y5ncu/SmolLM2-360M-Instruct-GLQ-block-diagonal-4bpw --quantization glq.

License

Apache 2.0. Original model: HuggingFaceTB/SmolLM2-360M-Instruct. Derivative quantization work; the base model and the GLQ tooling (github.com/cnygaard/glq) are both Apache 2.0.

---

🔗 GLQ on GitHub: https://github.com/cnygaard/glq — if you like it, a ⭐ is appreciated.