giannisdaras/olmo3-7b-tulu-1ksteps

Olmo3-7B-Tulu-3-SFT (1k steps)

Short-run supervised finetune of allenai/Olmo-3-1025-7B on allenai/tulu-3-sft-mixture for 1000 optimizer steps (~7% of one epoch). Used as a "warm teacher" in ambient knowledge distillation experiments — partially instruction-tuned, sitting between pretrained base and a fully SFT-d model.

Training Details

• Base model: allenai/Olmo-3-1025-7B (pretrained)
• Dataset: allenai/tulu-3-sft-mixture (~939k examples)
• Hardware: 32x NVIDIA GH200 (TACC Vista gh partition), flash-attn 2.x
• Framework: open-instruct + accelerate + DeepSpeed ZeRO-2
• Hyperparameters: max_seq_length 4096, per_device_bs 1, grad_acc 4 (effective batch 128), lr 5e-6 linear, warmup 0.03, weight_decay 0.0, bf16, gradient checkpointing on, tulu chat template, --add_bos, seed 123
• Steps: 1000 (vs ~14613 for a 2-epoch run)

Intended Use

Designed as a warm teacher for ambient knowledge distillation. The Llama-3.1-8B sibling (giannisdaras/tulu-vista-repro is the fully-SFT-d 14613-step Llama; the Llama warm teacher used a separate 1k Llama checkpoint) shows that 1k-step warm teachers can give modest gains over base-model cold teachers.

For instruction following, use a fully SFT-d Olmo3-Tulu instead.