aidendle94/GLM-5.2-NF4-Lloyd-Experts-GPTQ

Known defect: 5 experts are packed on the wrong grid

These weights are decoded per layer through the Lloyd codebook, but the quantizer selected its arm per expert. Five experts inside NF4-enabled layers were written with E2M1/MXFP4 nibbles and are therefore decoded against the wrong table at inference:

layer	expert	quantizer arm that won
5	164	r0 (E2M1)
8	86	best (E2M1)
9	15	best (E2M1)
32	30	rtn-fallback (E2M1)
46	139	rtn-fallback (E2M1)

Measured by decoding the real stored bytes both ways (L9/e15, 20 rows x 3 projections): 10.5% relative weight error introduced, coherent gain 0.9862, and 11.9% of that expert's weights are an exact-zero E2M1 code served as +/-0.2108 x scale.

A token hits at least one mis-packed expert with probability ~6.5%, and when it does, 1 of its 8 routed slots in 1 of 75 MoE layers is affected. Outputs stay coherent — this is degradation, not corruption — but it is a real defect.

Workaround: exclude layers 5, 8, 9, 32 and 46 from the NF4 layer list so they decode as E2M1, which is what their bytes actually are. (Layer 10 has the same problem and was already excluded upstream; the guard existed but missed these five.)

Also under investigation: reasoning length

Preliminary measurements suggest these weights may produce substantially longer chains of thought than the E2M1 baseline on agentic prompts. This is not yet a supported claim: at n=3 per arm the within-arm spread (2.35x) exceeded the apparent between-arm effect (2.12x), and an exact rank-sum test returns p=0.05, which is the floor achievable at that sample size, not evidence of a strong effect. Treat as an open question pending n>=10.

Note also that on direct weight-level metrics against BF16 ground truth this build still measures better than the E2M1 baseline (coherent gain 0.9944 vs 0.9903, relative RMS 10.80% vs 12.18% over 2.75M sampled weights).

GLM-5.2 — Lloyd-codebook NF4 routed experts (GPTQ)

Routed-expert weights for zai-org/GLM-5.2, quantised to 4 bits with a fitted Lloyd–Max codebook instead of the MXFP4/E2M1 grid, via GPTQ with activation-order and error feedback.

This repo contains experts only — the same shape as GLM-5.2-MXFP4-Experts-GPTQ. Attention, embeddings, norms and the shared expert come from the base model.

What this is

E2M1 (MXFP4) spends its 16 codes on a power-of-two grid {0,±.5,1,1.5,2,3,4,6}. Routed-expert weight distributions in this model are near-perfectly Gaussian (measured crest factor 2.366 vs 2.365 theoretical), so a codebook fitted to that distribution places its levels better for the same 4 bits.

Per-layer 8-magnitude codebooks were fitted by weighted Lloyd–Max over pooled |W / 2^e| ratios across all 256 experts × {gate, up, down}, weighted by (2^e)² so each ratio contributes in proportion to real weight-space MSE.

Measured against the E2M1 baseline on the same GPTQ pipeline: −16.8% expert reconstruction error, and Lloyd won on essentially every expert (19,194 / 19,200 in the original pass; 100% of experts in the top-up pass).

Contents

• L{n}.safetensors — per-layer expert weights, packed 4-bit + E8M0 block scales (weight, weight_scale; block 32 along K, low nibble = even K)
• grids/L{n}.json — that layer's 8 fitted magnitudes, {"<layer>": [m0..m7]}
• L{n}_metrics.json — per-expert reconstruction error, winning arm, offsets

Codes are indices into the per-layer grid — not E2M1

mag  = code & 0x07          # index into grids/L{n}.json magnitudes
sign = -1 if (code & 0x08) else +1
w    = sign * grid[mag] * 2.0 ** (scale_byte - 127)

A decoder that assumes the E2M1 value table will produce garbage. The grid is not amax-normalised: codes were packed against gmax * 2^e >= amax, so table[idx] * 2^(b-127) is the dequantisation.

Serving

Needs a kernel that decodes a 16-entry sign-symmetric codebook rather than the E2M1 bit-twiddle. A vLLM/b12x implementation is baked into aidendle94/sparkrun-vllm-ds4-gb10:production-hybrid-1.4 (nf4_e8m0_k32 source format / nf4_packed weight layout, per-layer opt-in via VLLM_NF4_LAYERS).

One implementation note that cost real debugging time: the packed nibble bytes are byte-identical to the E2M1 layout, so a staging path that dispatches on an exact weight_layout == "packed" comparison will silently route this format into a ModelOpt-swizzled loader and hand the dequant zeroed words for whole output-tile bands. Gate on "is this ModelOpt?", not on "is this packed?".

Coverage

All 75 MoE layers (3–77). Every layer ships with its fitted grid in grids/.

The format is per-layer independent, so partial adoption is valid — each converted layer is an independent improvement in weight fidelity.

Provenance

GPTQ on captured routed-expert activations (117 calibration shards/layer, ~240k tokens), damp=1.0 on rank-deficient per-expert Hessians, winner selected per expert against an E2M1 control arm with a both-arms guard, winner re-fit on 100% of that expert's tokens. Fitted and verified on RTX PRO 6000 / GB10.