lfm2-quantum-128m-sft (SFT checkpoint, step 849)

SFT (instruction-tuned) checkpoint of MarkChenX/lfm2-quantum-128m, an LFM2-style hybrid quantum GPT, trained with Quantum-GPT runs/run_lfm2_sft_rl.sh (Stage 3: SFT).

Unlike the base checkpoint, this model follows a chat format (single/multi turn conversation with <|user_start|>/<|assistant_start|> special tokens), not raw text continuation.

Architecture

Same architecture as the base model:

• 16 layers, hybrid mixer: 10 gated short-conv blocks + 6 GQA attention blocks (conv-first, pattern CCCCCCCCCCAAAAAA)
• n_embd=1024, 16 query heads / 8 KV heads (GQA), head_dim=64, seq_len=1024
• vocab_size=32768
• Quantum feed-forward network in every block (4 qubits, circuit depth 2)
• RoPE base theta=1e6, full (non-windowed) attention, value embeddings disabled

Full config: config.json (metadata only) and meta_000849.json (exact training config this checkpoint was produced with). This is a custom architecture, not a transformers model -- there is no AutoModel support.

SFT details

• Warm-started from the base model's optimizer state (momentum buffers), fine-tuned on a mixture of SmolTalk (general chat), MMLU auxiliary_train (x3 epochs, teaches multiple choice), and GSM8K train (x4 epochs, teaches math/tool use)
• Validation bpb at step 849: 0.4337 (vs. 0.92 for the base model on its own pretraining val set -- different eval sets, not directly comparable)

Inference

This repo bundles the minimal nanochat/ source needed to load and run the model, so it's self-contained (no need to clone the full training repo).

pip install torch tiktoken rustbpe filelock kernels
hf download MarkChenX/lfm2-quantum-128m-sft --local-dir ./lfm2-quantum-128m-sft
cd lfm2-quantum-128m-sft
python inference.py --prompt "What is the capital of France?"

See inference.py for the full loading + generation code: it builds the model from model_000849.pt + meta_000849.json, loads the tokenizer from tokenizer/, wraps the prompt in the chat special tokens (<|user_start|>...<|user_end|><|assistant_start|>), and streams the response via the model's own .generate() until <|assistant_end|>.

Resuming SFT / continuing to RL

optim_000849_rank0.pt is the matching Muon/AdamW optimizer state for this step. To continue training with the original Quantum-GPT repo, place model_000849.pt, optim_000849_rank0.pt and meta_000849.json under $NANOCHAT_BASE_DIR/chatsft_checkpoints/lfm2-quantum-128m/, then run RL:

MODEL_TAG=lfm2-quantum-128m SKIP_SFT=1 bash runs/run_lfm2_sft_rl.sh