lfm2-quantum-128m (base checkpoint, step 2162)

LFM2-style hybrid quantum GPT, budget tier, trained with Quantum-GPT runs/run_lfm2_quantum_mini.sh.

This is a base (pretrained, not instruction-tuned) checkpoint at step 2162 (final step of pretraining) -- it does raw text continuation, not chat.

Architecture

• 16 layers, hybrid mixer: 10 gated short-conv blocks + 6 GQA attention blocks (conv-first, pattern CCCCCCCCCCAAAAAA)
• n_embd=1024, 16 query heads / 8 KV heads (GQA), head_dim=64, seq_len=1024
• vocab_size=32768
• Quantum feed-forward network in every block (4 qubits, circuit depth 2, simulated exactly in PyTorch -- see nanochat/gpt.py's QuantumMLP)
• RoPE base theta=1e6, full (non-windowed) attention (window_pattern=L)
• Value embeddings disabled

Full config: config.json (metadata only) and meta_002162.json (exact training config this checkpoint was produced with).

This is a custom architecture, not a transformers model -- there is no AutoModel support. config.json is provided for discoverability/metadata only; to actually load the model, use the bundled nanochat/ package as shown below.

Inference

This repo bundles the minimal nanochat/ source needed to load and run the model, so it's self-contained (no need to clone the full training repo).

pip install torch tiktoken rustbpe filelock kernels
# download this repo, e.g.:
hf download MarkChenX/lfm2-quantum-128m --local-dir ./lfm2-quantum-128m
cd lfm2-quantum-128m
python inference.py --prompt "The history of quantum computing"

See inference.py for the full loading + generation code (~15 lines): it calls nanochat.checkpoint_manager.build_model(".", step=2162, ...) to build the model from model_002162.pt + meta_002162.json, loads the tokenizer from tokenizer/, and streams tokens via the model's own .generate().

Resuming pretraining

optim_002162_rank0.pt is the matching Muon/AdamW optimizer state (momentum buffers etc.) for this step. To continue pretraining with the original Quantum-GPT repo, place model_002162.pt, optim_002162_rank0.pt and meta_002162.json under $NANOCHAT_BASE_DIR/base_checkpoints/lfm2-quantum-128m/, then run:

RESUME_FROM_STEP=2162 MODEL_TAG=lfm2-quantum-128m bash runs/run_lfm2_quantum_mini.sh

For SFT/RL instead of continued pretraining, only model_002162.pt + meta_002162.json + tokenizer/ are needed (the optimizer shard is pretraining-only).