---
license: apache-2.0
language:
- en
library_name: pytorch
pipeline_tag: text-generation
tags:
- recurrent
- peer
- mixture-of-experts
- product-quantization
- cpu
- compression
- research
---

# DNA-DiskChat-2B-PEER-v26

Recurrent PEER controller (1.999B logical params) with a **product-quantized deployment checkpoint that passes both size and quality gates** — the fix for the v25 campaign's compression failure.

| Field | Value |
|---|---:|
| Logical parameters | 1,999,254,018 |
| Active parameters / token | 40,189,952 |
| Deployment checkpoint | **176.6 MiB** (`controller.v26q`) |
| Effective bits / parameter | **0.741** |
| Compression validation CE delta | **-0.028** (v25: +4.95) |
| Compression quality gate (<= +0.01) | ✅ **PASS** |
| 195 MiB size gate | ✅ **PASS** |
| Architecture | Recurrent (Linear Attention `diag_scan`) + PEER (product-key MoE, 1.89M experts, top-8) |

This is a **research artifact**: the win here is the compression codec, not code-generation capability (see [Benchmarks](#benchmarks)).

## What changed vs v25

v25 quantized *every* tensor to ternary {-1, 0, +1} with 88% forced zeros and two per-row means. The PEER expert pool (1.89M x 512 fp32 lookup vectors) is used as exact additive vectors in the hidden state; ternary + 88% sparsity destroyed the per-value magnitude and validation CE went 5.25 → 10.20 (**+4.95**). A "less sparsity" retry made it worse (+6.80), proving ternary sign-means cannot represent these vectors at any density.

v26 replaces that with:

| Component | v25 (fail) | v26 (pass) |
|---|---|---|
| pool.up / pool.down (1.89M x 512 x 2) | ternary, 88% zero | **Product Quantization**: 8 groups x 64-dim blocks, 12-bit centroids, fp16 codebooks |
| dense weights | ternary everywhere | fp16 |
| recurrent `proj`/`o`, layers 0-3 | ternary | raw fp32 (fp16-sensitive, see below) |

**Why fp32 for exactly layers 0-3:** a per-layer ablation (each layer's `proj`+`o` replaced by fp16, one at a time) isolated the fp16 sensitivity:

| layer | fp16 CE delta | layer | fp16 CE delta |
|---|---|---|---|
| 0 | **+0.0443** | 5 | +0.0003 |
| 1 | +0.0073 | 6 | -0.0101 |
| 2 | +0.0026 | 7 | -0.0102 |
| 3 | +0.0054 | 8 | -0.0052 |
| 4 | -0.0041 | | |

Early recurrent projections (fed into the 512-step `diag_scan` recurrence) are fp16-sensitive; later layers are not. Storing layers 0-3 in fp32 costs ~10.5 MB and removes the entire residual.

## Measured compression results

Validation: 8 batches x 4 x 512 tokens from a deterministic fineweb-edu tail (docs 2000:2199, sha256 `7e30af1a4412d2fe990294d0dfb158bbd987ecbc6e707af4bdd30e1b816e7568`), bf16 autocast, greedy.

| Metric | v25 (ternary) | v26 all-fp16 | **v26 mixed (final)** |
|---|---|---|---|
| File size | 186.6 MiB | 166.6 MiB | **176.6 MiB** |
| Effective bits/param | 0.783 | 0.699 | **0.741** |
| Under 195 MiB | ✅ | ✅ | ✅ |
| Base validation CE | 5.247* | 5.475 | **5.475** |
| Compressed CE | 10.199 | 5.501 | **5.448** |
| CE delta | **+4.952 ❌** | +0.0255 | **-0.0276 ✅** |

\* v25 base CE used a different, unpublished validation set; v26 numbers are on the reproducible set above. The negative delta is within noise — the pool is effectively lossless at B12 (dense-only ablation +0.0255 vs full -0.0276 means the pool contributes ~0).

![Compression quality](charts/compression_ce_delta.png)
![Checkpoint size](charts/checkpoint_size.png)

## Codec (V26PQ01)

- **Pool payload:** 53.8 MB for 1.938B pool params = **0.222 bits/param**. Codebooks: 2 tables x 8 groups x 4096 x 64 dims fp16 (8.4 MB); indices: 2 x 1.89M x 8 x 12 bits, LSB-packed (45.5 MB).
- **Dense:** fp16 section 110.4 MB; fp32 (layers 0-3 proj/o) 21.0 MB; metadata 11.6 KB.
- Format: `header | manifest | PQ payload (crc'd, B-bit packed) | fp16 section | fp32 section | sha256 footer`. Reader is backward compatible with the all-fp16 v1 layout. PQ state is reusable across exports (`--pq-state-from`), so the ~20 min k-means runs once.
- The pool is near-isotropic (flat PCA spectrum, std exactly 0.02 = init): experts were barely trained and SFT froze the pool. PQ is the right tool precisely because it preserves per-value magnitude without needing low-rank structure.

## Training throughput

Same architecture and training loop as v25 (10B base tokens, then code SFT):

![Training throughput](charts/training_tok_s.png)

| Metric | Value |
|---|---:|
| Mean tokens/s (base) | 55,504 |
| Median tokens/s (base) | 55,656 |
| p05 / p95 | 51,777 / 60,148 |
| Measured windows | 24,611 |

## Benchmarks

EvalPlus 0.3.1, greedy pass@1, on the SFT checkpoint (not the compressed file). Peer numbers are **web-sourced** via Brave searches (patchright, logged in `search_results/`) and cite the model's official reports; ours are measured locally on the same harness.

![Benchmark comparison](charts/benchmark_comparison.png)

| Model | Size | HumanEval pass@1 | MBPP pass@1 |
|---|---:|---:|---:|
| Phi-2 | 2.7B | 59.2% | 59.1% |
| Qwen2.5-Coder-1.5B | 1.5B | 43.3% | 50.0% |
| SmolLM2-1.7B | 1.7B | 28.1% | n/a |
| Gemma-2 2B | 2B | 17.7% | n/a |
| TinyLlama-1.1B | 1.1B | 6.5-8.5% | 12.3% |
| **DNA-DiskChat-2B-PEER-v26** | **2B (40M active)** | **0.0%** | **0.0%** |

**Honest position:** the v26 model scores 0% on HumanEval/MBPP. The v25 eval generated solutions that reproduce signatures and docstrings correctly but with nonsense bodies (`def is_element(n): return 0`, docstring-echo loops). The model learned code *format* on ~4k CodeAlpaca SFT steps but not code *semantics* at ~40M active params/token, after only 10B pretrain tokens. The compression fix in v26 does not change this; the two are independent problems.

Fix path (documented, not yet run): higher-quality verified SFT data (20-30k records, not CodeAlpaca), unfreeze the PEER pool during SFT (`--train-pool`; v25 SFT froze it so the 1.9B logical params contributed nothing), and longer/more diverse pretraining. Re-run with the same pinned EvalPlus 0.3.1 harness and report honestly.

## Deployment

- `deployment/controller.v26q` — 176.6 MiB V26PQ01 checkpoint (PQ pool + fp16/fp32 dense).
- `tokenizer.json` — 32k BPE (FineWeb).
- `data/validation.u16` — reproducible validation tail (8 x 4 x 513 tokens, fineweb-edu docs 2000:2199).
- Reader/decoder: `scripts/v26_pq/pq_page_format.py` (`CheckpointReader.tensor(name)`).
- External codon table (`codons.u8`, 512 MiB, not part of the 195 MiB controller budget) is unchanged from the same checkpoint and available at [jaivial/dna-diskchat-2b-peer-v25](https://huggingface.co/jaivial/dna-diskchat-2b-peer-v25/blob/main/deployment/codons.u8).

## Reproducibility

- Code: `scripts/v26_pq/` (pq_codec.py, pq_page_format.py, export_pq.py, evaluate_pq.py, make_validation.py, inspect_ckpt.py, sweep_dense.py, test_pq.py — 8/8 unit tests pass).
- Validation set: docs 2000:2199 of `HuggingFaceFW/fineweb-edu/sample-100BT`, 16,416 tokens, sha256 `7e30af1a4412d2fe990294d0dfb158bbd987ecbc6e707af4bdd30e1b816e7568`.
- All experiments ran on a Vast AI instance (RTX 3090, 51 GB RAM); no results fabricated.
- The v25 model card and results remain at [jaivial/dna-diskchat-2b-peer-v25](https://huggingface.co/jaivial/dna-diskchat-2b-peer-v25).

## Intended use

Research on compact recurrent codecs, CPU/disk training, and low-bit deployment of MoE controllers. Not production software. Generated output is untrusted.