# v26 — PQ codec fixes the v25 compression failure

**Date:** 2026-08-02
**Status:** campaign complete. Both v25 gates now PASS. Compression validation CE delta **-0.028** (was **+4.95**), checkpoint **176.6 MiB** (was 186.6 MiB). EvalPlus unchanged at 0% — that failure is a model-capability gap, not a codec issue (see below).
**Where:** experiments run on Vast AI instance 46602403 (RTX 3090, 51 GB RAM). No local experiment runs.

## Why v25 failed

Two independent failures, both confirmed by measured ablation, not inferred:

1. **Compression gate fail (+4.95 CE).** The v25 export quantized *every* floating tensor to ternary {-1, 0, +1} with 88% forced zeros and two per-row fp16 means. The PEER expert pool (1.894M x 512 x 2, fp32-trained lookup embeddings) is used as exact additive vectors in the hidden state. Ternary + 88% sparsity destroys the per-value magnitude, so the pooled memory contribution loses its shape. Validation CE went 5.247 → 10.199. A "less sparsity" retry (zero_fraction 0.85) made it *worse* (+6.80), confirming ternary sign-means cannot represent these vectors at any density.

2. **EvalPlus fail (0/164, 0/378).** The eval ran on the full-precision SFT checkpoint, not the compressed file. The generated solutions reproduce signatures/docstrings correctly but the bodies are nonsense (`def is_element(n): return 0`, docstring-echo loops). The model learned code *format* on 4k CodeAlpaca steps but not code *semantics* at ~40M active params/token. This is a training-data/capacity gap, not an adapter or quantization bug. The v26 codec does not and cannot fix it.

## The v26 fix

Product quantization for the pool (the standard codec for embedding tables), fp16 for dense weights, fp32 for the dense tensors that are fp16-sensitive:

| Component | v25 (fail) | v26 (pass) |
|---|---|---|
| pool.up/down (1.894M x 512 x 2) | ternary, 88% zero | PQ: 8 groups x 64-dim blocks, 12-bit centroids, fp16 codebooks |
| dense weights | ternary everywhere | fp16 |
| proj/o layers 0-3 (recurrent) | ternary | raw fp32 |
| external codon table | 512 MiB (excluded) | unchanged |

### Why fp32 for exactly layers 0-3

A per-layer ablation (each layer's proj+o replaced by fp16, one at a time) isolated the fp16-rounding sensitivity:

| layer | fp16 CE delta |
|---|---|
| 0 | **+0.0443** |
| 1 | +0.0073 |
| 2 | +0.0026 |
| 3 | +0.0054 |
| 4 | -0.0041 |
| 5 | +0.0003 |
| 6 | -0.0101 |
| 7 | -0.0102 |
| 8 | -0.0052 |

Early recurrent projections (fed into the 512-step `diag_scan` recurrence) are fp16-sensitive; later layers are not (their small errors act as negligible noise/regularization). Storing layers 0-3 proj/o in fp32 costs only ~10.5 MB and removes the entire +0.025 residual.

## Results (measured on box, 8 x 4 x 512 validation batches)

| Metric | v25 (ternary) | v26 all-fp16 | v26 mixed (final) |
|---|---|---|---|
| File size | 186.6 MiB | 166.6 MiB | **176.6 MiB** |
| Effective bits/param | 0.783 | 0.699 | **0.741** |
| Under 195 MiB | true | true | **true** |
| Base validation CE | 5.247* | 5.475 | **5.475** |
| Compressed CE | 10.199 | 5.501 | **5.448** |
| CE delta | **+4.952** | +0.0255 | **-0.0276** |
| Quality gate (<= +0.01) | **false** | false | **true** |

\* v25 base CE used a different (unreproducible) validation set. v26 uses a fresh deterministic set: docs 2000:2199 of `HuggingFaceFW/fineweb-edu/sample-100BT`, sha256 `7e30af1a4412d2fe990294d0dfb158bbd987ecbc6e707af4bdd30e1b816e7568`, 16,416 tokens (8x4x513), tokenizer from the published repo. The negative delta is within noise; the pool is effectively lossless at B12 (dense-only ablation +0.0255 vs full -0.0276: pool contributes ~0).

## Codec notes

- Pool payload: 53.8 MB for 1.938B pool params = 0.222 bits/param. Codebooks: 2 tables x 8 groups x 4096 x 64 dims x fp16 = 8.4 MB; indices: 2 x 1.894M x 8 x 12 bits packed = 45.5 MB.
- Dense fp16 section: 110.4 MB. Dense fp32 (layers 0-3 proj/o): 21.0 MB. Metadata: 11.6 KB.
- Format `V26PQ01` version 2: header | manifest | PQ payload (crc'd, B-bit packed) | fp16 section | fp32 section | sha256 footer. Reader is backward compatible with v1 (all-fp16) files. Pool PQ state is reusable across exports (`--pq-state-from`), so the 20-minute k-means runs once.
- The v25 pool is near-isotropic (PCA spectrum flat, std exactly 0.02 = init): experts were barely trained and SFT froze the pool. PQ is the right tool precisely because it preserves per-value magnitude without needing low-rank structure.

## EvalPlus status and fix path

EvalPlus remains 0%. The honest position: compression is fixed; the model cannot write working code. Fixes, in order of expected value:
1. Better SFT data (CodeAlpaca is auto-generated and noisy; use verified instruction sets, more answer diversity, 20-30k records per the v25 eval README, deterministic holdout).
2. Unfreeze the pool during SFT (`--train-pool`); the v25 SFT froze it, so the 1.9B logical params contributed nothing to code adaptation.
3. Longer/more diverse pretraining before SFT if the base model lacks code semantics entirely.
4. Re-run EvalPlus with the same pinned EvalPlus 0.3.1 harness. Report honestly.

## Reproducibility

- Code: `dna-2b-tests/v26_pq/` (pq_codec.py, pq_page_format.py, export_pq.py, evaluate_pq.py, make_validation.py, inspect_ckpt.py, sweep_dense.py, test_pq.py). All 8 unit tests pass.
- Artifacts: `controller_v26.v26q` (185,203,022 bytes), validation set hash above.
- Vast instance 46602403 was used for all experiments and destroyed after results were pulled.