jwl50-stanford/gdn-paper-120m-fixed-qk128-v64-fp32res-gpt2init-20k-r1

Fixed-State GDN Paper 120M, Q/K Head Dim 128, V Head Dim 64

This repository contains the final PyTorch Lightning checkpoint for the fixed-state Gated DeltaNet language-model experiment m06d10y26-gdn-paper-qk128-v64-fp32res-gpt2init-20k-r1.

The model has 12 layers, hidden size 768, and 12 heads. Each head uses a fixed Q/K feature dimension of 128 and a V dimension of 64. This is an ordinary fixed-state GDN: it does not use a growing-state mask or an expansion schedule.

Files

• last.ckpt: final checkpoint, byte-identical to the Perlmutter step_20000.ckpt.
• train_config.yaml: exact training config preserved in the archived GDN branch.
• eval_metrics.json: training/evaluation provenance, metrics, and the checkpoint SHA-256 digest.

Provenance

• Training run: m06d10y26-gdn-paper-qk128-v64-fp32res-gpt2init-20k-r1.
• Training Slurm job: 54260616.
• Training source commit recorded by W&B: 9b40978e08d174bb24b1c675cdc1d2a655996ab4.
• Preserved config/code branch: HazyResearch/mlp-mixer commit d2b24d685bc12f6e974d3ce68ffe2563a477cea1.
• W&B: hazy-research/mlp-mixer.

Architecture

• Hidden size: 768
• Layers: 12
• Heads: 12
• Q/K head dimension: 128
• V head dimension: 64 (expand_v=0.5)
• Independent Q/K projections: 1
• Residual stream: FP32
• Residual-output initialization: GPT-2 normal
• Recurrent state: fixed-size

Metrics

• Full Pile validation PPL from the finished W&B run: 10.2112963416.
• Full Pile validation loss: 2.3234939575 over 7,667,089,408 tokens.
• Independent Pile validation 10% PPL at sequence length 2048: 10.1007396226.
• Rare-AR overall PPL at sequence length 2048: 10.1014004483.
• Rare-AR selected accuracy: 0.708458599321 over 1,156,539 selected event tokens.

Usage Note

This is a Lightning checkpoint for the custom mlp-mixer / Gated DeltaNet training and evaluation stack, not a standalone Transformers-format model. Use a compatible checkout of the preserved GDN branch and load last.ckpt through that stack. The repository's existing checkpoint downloader convention expects this filename.