Mahesh111000/hanabi-qwen3-32b-lora-step50

Hanabi Qwen3-32B LoRA — step 50 (RL / GRPO via prime-rl)

RL training state for resuming on another machine. Base model: Qwen/Qwen3-32B (bf16).

Note: 97.5% of checkpoint_step_50/ is the frozen base model stored as fp32 master weights (32.76B x 4 bytes = 131 GB). Only ~3.2 GB is unique. If you don't need the exact optimizer state and step counter, just use adapter/ (1.07 GB) with a fresh optimizer — it holds all the learning, and it lets you change the learning rate (a resume does not).

Contents

path	size	purpose
adapter/	1.1 GB	LoRA adapter (r=32, alpha=64). The entire trained delta. PEFT/vLLM loadable.
checkpoint_step_50/	126 GB	prime-rl DCP checkpoint — model + optimizer + scheduler, exact resume.
code/	small	the custom reward env, config, and launcher (not on PyPI — required).

Training state at step 50

• Checkpoint saved after step 49 completed; resuming runs step 50 next.
• LoRA r=32, alpha=64, lr 1e-5, all 7 linear projections.
• batch 256 = 16 problems x 16 rollouts, max_tokens 26624, seq_len 32768.
• GPUs: 4 inference (TP=4) + 4 trainer (cp=4), 8x A100-SXM4-80GB.
• Reward = deduction_score + move_reward (range [0,2]).

Observed through step 101

Train reward (256 rollouts/step, so SEM ~0.02 — this metric is resolvable):

window	mean train reward
steps 0–50	~1.28
steps 67–81	1.499
steps 82–101	1.557

Recent steps: 95 1.6600 · 96 1.5811 · 97 1.5674 · 98 1.4306 · 99 1.6806 · 100 1.5396 · 101 1.6122. Grad norm ~0.0006, entropy ~0.578, mismatch KL ~0.005.

Train reward has genuinely risen (~1.28 → ~1.56, roughly 15 SEM). Mean completion length grew from ~6,145 tokens (base model) to ~9,865, i.e. the policy reasons ~1.6x longer.

Held-out val, however, is flat at baseline:

step	0	70	80	90	100
val reward	1.2401	1.3656	1.1854	1.2400	1.2669

Read this metric with care — and do not over-read it in either direction. [orchestrator.val] is num_examples = 32, rollouts_per_example = 1, temperature unset (server default, i.e. stochastic). So each point is 32 one-shot samples, giving a noise band of roughly ±0.12 at 2σ — every reading above overlaps every other one and the baseline. A 32-sample stochastic eval cannot resolve the size of change a LoRA run produces. It is not evidence of learning, and it is not evidence of failure; it is under-powered.

To actually settle generalization, run an offline eval instead: all 64 held-out examples x 8 rollouts (~512 samples, SEM ~0.02) on the base model vs this checkpoint, using the same rubric.

is_truncated = 0.0 throughout, so nothing is hitting the 26,624-token cap. copy_rate ~0.55 and flat, i.e. the degenerate "copy the previous belief state" policy is not being learned. Entropy drifts down (0.60 → 0.578) and mismatch KL up (0.0029 → 0.005), both consistent with a policy slowly concentrating — worth watching, not yet a problem.

The small grad norm is expected, not a fault: the loss is token-mean-reduced over ~3.9M tokens per step and advantages are raw within-group deviations (no std normalisation), while Adam is scale-invariant so updates remain ~lr-sized.

Resume on a new server

git clone https://github.com/PrimeIntellect-ai/prime-rl.git
cd prime-rl && git checkout 16e747c25e81c5c90bd860c68a588f7b151353a6
uv sync

huggingface-cli download Mahesh111000/hanabi-qwen3-32b-lora-step50 --local-dir ./hanabi-state

# custom reward env (required — not on PyPI)
uv pip install --python .venv/bin/python --no-deps -e ./hanabi-state/code/hanabi-deduction

cp ./hanabi-state/code/rl_32b.toml configs/hanabi_deduction/
mkdir -p outputs/checkpoints
cp -r ./hanabi-state/checkpoint_step_50 outputs/checkpoints/step_50

VLLM_ATTENTION_BACKEND=FLASHINFER \
  uv run rl @ configs/hanabi_deduction/rl_32b.toml --ckpt.resume-step 50

Dataset: Mahesh111000/Hanabi-init-30turns (or point dataset_path at a local copy).

Expect: the first step after any (re)start is a cold start while the async pipeline refills (Async Level climbs from -2 to 0) — measured 148 min, then 85, then settling to ~60–92 min per step. The trainer needs ~40 min and hides entirely inside the rollout phase, so step time is inference-bound. Needs 8x80 GB GPUs.

VLLM_ATTENTION_BACKEND=FLASHINFER in the command above is not cosmetic — it is the single biggest win found, measured over full training steps on 8x A100-SXM4-80GB:

backend	steps	mean min/step
FlashAttention	52–66	124.5
FlashInfer	67–81	81.6
FlashInfer	67–101 (35 steps)	91.1

~1.37x end-to-end, measured over 35 steps. Note the earlier step-65 README quoted 1.56x from the first 15 FlashInfer steps (81.6 min); with 20 more steps the mean regressed to 91.1 min, so 1.37x is the honest figure and the 15-step window was optimistic. A standalone decode benchmark predicted 1.26x, so a modest overlap benefit is real, just smaller than first measured. Since decode here is KV-bandwidth-bound (256 KiB/token, ~5,400-token must-attend prompt = ~47% of KV), attention-kernel efficiency dominates step time.

Full code, configs and profiling tooling: https://github.com/kaousheik-26/hanabi_latest_train_32b

Gotchas baked into the config (learned the hard way)

• dist_timeout_seconds = 18000 — the default 600s kills the trainer on the cold-start step.
• [orchestrator.client] timeout = 7200 — the default 1200s silently drops long rollouts.
• ckpt.keep_last = 2 — without it, checkpoints fill the disk (~126 GB per save).
• [trainer.tokenizer] name pinned — prime-rl otherwise resolves it from a stale default.
• Resume always inherits the checkpoint's LR; the config's lr is ignored on resume. To change the learning rate you must start fresh from merged weights, not resume.
• prime-rl's outputs/weights/ are NOT merged (bitwise identical to base) — don't rely on them. clean_lora_state_dict() (trainer/lora.py:220) discards the adapter. Use merge_lora.py.
• A checkpoint named step_N is the state to resume at N, i.e. saved after step N-1 finished.
• vLLM prefix caching is inert here: verifiers sets prompt_logprobs=True, which makes vLLM set skip_reading_prefix_cache and bypass the cache entirely (0 queries, not 0 hits).