DNA-DiskChat-2B-PEER-v26
Recurrent PEER controller (1.999B logical params) with a product-quantized deployment checkpoint that passes both size and quality gates — the fix for the v25 campaign's compression failure.
| Field | Value |
|---|---|
| Logical parameters | 1,999,254,018 |
| Active parameters / token | 40,189,952 |
| Deployment checkpoint | 176.6 MiB (controller.v26q) |
| Effective bits / parameter | 0.741 |
| Compression validation CE delta | -0.028 (v25: +4.95) |
| Compression quality gate (<= +0.01) | ✅ PASS |
| 195 MiB size gate | ✅ PASS |
| Architecture | Recurrent (Linear Attention diag_scan) + PEER (product-key MoE, 1.89M experts, top-8) |
This is a research artifact: the win here is the compression codec, not code-generation capability (see Benchmarks).
What changed vs v25
v25 quantized every tensor to ternary {-1, 0, +1} with 88% forced zeros and two per-row means. The PEER expert pool (1.89M x 512 fp32 lookup vectors) is used as exact additive vectors in the hidden state; ternary + 88% sparsity destroyed the per-value magnitude and validation CE went 5.25 → 10.20 (+4.95). A "less sparsity" retry made it worse (+6.80), proving ternary sign-means cannot represent these vectors at any density.
v26 replaces that with:
| Component | v25 (fail) | v26 (pass) |
|---|---|---|
| pool.up / pool.down (1.89M x 512 x 2) | ternary, 88% zero | Product Quantization: 8 groups x 64-dim blocks, 12-bit centroids, fp16 codebooks |
| dense weights | ternary everywhere | fp16 |
recurrent proj/o, layers 0-3 |
ternary | raw fp32 (fp16-sensitive, see below) |
Why fp32 for exactly layers 0-3: a per-layer ablation (each layer's proj+o replaced by fp16, one at a time) isolated the fp16 sensitivity:
| layer | fp16 CE delta | layer | fp16 CE delta |
|---|---|---|---|
| 0 | +0.0443 | 5 | +0.0003 |
| 1 | +0.0073 | 6 | -0.0101 |
| 2 | +0.0026 | 7 | -0.0102 |
| 3 | +0.0054 | 8 | -0.0052 |
| 4 | -0.0041 |
Early recurrent projections (fed into the 512-step diag_scan recurrence) are fp16-sensitive; later layers are not. Storing layers 0-3 in fp32 costs ~10.5 MB and removes the entire residual.
Measured compression results
Validation: 8 batches x 4 x 512 tokens from a deterministic fineweb-edu tail (docs 2000:2199, sha256 7e30af1a4412d2fe990294d0dfb158bbd987ecbc6e707af4bdd30e1b816e7568), bf16 autocast, greedy.
| Metric | v25 (ternary) | v26 all-fp16 | v26 mixed (final) |
|---|---|---|---|
| File size | 186.6 MiB | 166.6 MiB | 176.6 MiB |
| Effective bits/param | 0.783 | 0.699 | 0.741 |
| Under 195 MiB | ✅ | ✅ | ✅ |
| Base validation CE | 5.247* | 5.475 | 5.475 |
| Compressed CE | 10.199 | 5.501 | 5.448 |
| CE delta | +4.952 ❌ | +0.0255 | -0.0276 ✅ |
* v25 base CE used a different, unpublished validation set; v26 numbers are on the reproducible set above. The negative delta is within noise — the pool is effectively lossless at B12 (dense-only ablation +0.0255 vs full -0.0276 means the pool contributes ~0).
Codec (V26PQ01)
- Pool payload: 53.8 MB for 1.938B pool params = 0.222 bits/param. Codebooks: 2 tables x 8 groups x 4096 x 64 dims fp16 (8.4 MB); indices: 2 x 1.89M x 8 x 12 bits, LSB-packed (45.5 MB).
- Dense: fp16 section 110.4 MB; fp32 (layers 0-3 proj/o) 21.0 MB; metadata 11.6 KB.
- Format:
header | manifest | PQ payload (crc'd, B-bit packed) | fp16 section | fp32 section | sha256 footer. Reader is backward compatible with the all-fp16 v1 layout. PQ state is reusable across exports (--pq-state-from), so the ~20 min k-means runs once. - The pool is near-isotropic (flat PCA spectrum, std exactly 0.02 = init): experts were barely trained and SFT froze the pool. PQ is the right tool precisely because it preserves per-value magnitude without needing low-rank structure.
Training throughput
Same architecture and training loop as v25 (10B base tokens, then code SFT):
| Metric | Value |
|---|---|
| Mean tokens/s (base) | 55,504 |
| Median tokens/s (base) | 55,656 |
| p05 / p95 | 51,777 / 60,148 |
| Measured windows | 24,611 |
Benchmarks
EvalPlus 0.3.1, greedy pass@1, on the SFT checkpoint (not the compressed file). Peer numbers are web-sourced via Brave searches (patchright, logged in search_results/) and cite the model's official reports; ours are measured locally on the same harness.
| Model | Size | HumanEval pass@1 | MBPP pass@1 |
|---|---|---|---|
| Phi-2 | 2.7B | 59.2% | 59.1% |
| Qwen2.5-Coder-1.5B | 1.5B | 43.3% | 50.0% |
| SmolLM2-1.7B | 1.7B | 28.1% | n/a |
| Gemma-2 2B | 2B | 17.7% | n/a |
| TinyLlama-1.1B | 1.1B | 6.5-8.5% | 12.3% |
| DNA-DiskChat-2B-PEER-v26 | 2B (40M active) | 0.0% | 0.0% |
Honest position: the v26 model scores 0% on HumanEval/MBPP. The v25 eval generated solutions that reproduce signatures and docstrings correctly but with nonsense bodies (def is_element(n): return 0, docstring-echo loops). The model learned code format on ~4k CodeAlpaca SFT steps but not code semantics at ~40M active params/token, after only 10B pretrain tokens. The compression fix in v26 does not change this; the two are independent problems.
Fix path (documented, not yet run): higher-quality verified SFT data (20-30k records, not CodeAlpaca), unfreeze the PEER pool during SFT (--train-pool; v25 SFT froze it so the 1.9B logical params contributed nothing), and longer/more diverse pretraining. Re-run with the same pinned EvalPlus 0.3.1 harness and report honestly.
Deployment
deployment/controller.v26q— 176.6 MiB V26PQ01 checkpoint (PQ pool + fp16/fp32 dense).tokenizer.json— 32k BPE (FineWeb).data/validation.u16— reproducible validation tail (8 x 4 x 513 tokens, fineweb-edu docs 2000:2199).- Reader/decoder:
scripts/v26_pq/pq_page_format.py(CheckpointReader.tensor(name)). - External codon table (
codons.u8, 512 MiB, not part of the 195 MiB controller budget) is unchanged from the same checkpoint and available at jaivial/dna-diskchat-2b-peer-v25.
Reproducibility
- Code:
scripts/v26_pq/(pq_codec.py, pq_page_format.py, export_pq.py, evaluate_pq.py, make_validation.py, inspect_ckpt.py, sweep_dense.py, test_pq.py — 8/8 unit tests pass). - Validation set: docs 2000:2199 of
HuggingFaceFW/fineweb-edu/sample-100BT, 16,416 tokens, sha2567e30af1a4412d2fe990294d0dfb158bbd987ecbc6e707af4bdd30e1b816e7568. - All experiments ran on a Vast AI instance (RTX 3090, 51 GB RAM); no results fabricated.
- The v25 model card and results remain at jaivial/dna-diskchat-2b-peer-v25.
Intended use
Research on compact recurrent codecs, CPU/disk training, and low-bit deployment of MoE controllers. Not production software. Generated output is untrusted.