---
license: apache-2.0
language:
- en
pipeline_tag: text-generation
tags:
- olmo3
- tulu-3
- sft
- 1k-steps
base_model: allenai/Olmo-3-1025-7B
datasets:
- allenai/tulu-3-sft-mixture
---

# Olmo3-7B-Tulu-3-SFT (1k steps)

Short-run supervised finetune of `allenai/Olmo-3-1025-7B` on `allenai/tulu-3-sft-mixture` for **1000 optimizer steps** (~7% of one epoch). Used as a "warm teacher" in ambient knowledge distillation experiments — partially instruction-tuned, sitting between pretrained base and a fully SFT-d model.

## Training Details

- **Base model**: `allenai/Olmo-3-1025-7B` (pretrained)
- **Dataset**: `allenai/tulu-3-sft-mixture` (~939k examples)
- **Hardware**: 32x NVIDIA GH200 (TACC Vista gh partition), flash-attn 2.x
- **Framework**: open-instruct + accelerate + DeepSpeed ZeRO-2
- **Hyperparameters**: max_seq_length 4096, per_device_bs 1, grad_acc 4 (effective batch 128), lr 5e-6 linear, warmup 0.03, weight_decay 0.0, bf16, gradient checkpointing on, tulu chat template, `--add_bos`, seed 123
- **Steps**: 1000 (vs ~14613 for a 2-epoch run)

## Intended Use

Designed as a warm teacher for ambient knowledge distillation. The Llama-3.1-8B sibling (`giannisdaras/tulu-vista-repro` is the fully-SFT-d 14613-step Llama; the Llama warm teacher used a separate 1k Llama checkpoint) shows that 1k-step warm teachers can give modest gains over base-model cold teachers.

For instruction following, use a fully SFT-d Olmo3-Tulu instead.