sashaboguraev/pythia-160m-ppt-random_numbers_steps100-seed1024

Pythia-160M Pre-Pretraining: random_numbers_steps100 (seed 1024)

Trained from scratch using the ppt pre-pretraining research framework.

Training Details

Parameter	Value
Base architecture	EleutherAI/pythia-160m (reinitialized)
Regimen	random_numbers_steps100
Seed	1024
Stage 1 dataset	Random number sequences
Stage 1 steps	100
Stage 2 dataset	OpenWebText
Stage 2 steps	10000
Optimizer	AdamW (lr=1e-3, wd=0.0)
Effective batch size	64
Sequence length	2048

Control Design

Stage 1: Random number sequences (unstructured baseline).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("sashaboguraev/pythia-160m-ppt-random_numbers_steps100-seed1024")
tokenizer = AutoTokenizer.from_pretrained("sashaboguraev/pythia-160m-ppt-random_numbers_steps100-seed1024")

Citation

If you use this model, please cite the original pre-pretraining papers:

• Papadimitriou & Jurafsky (2020) — tilt-transfer
• Hahn & Rofin (2024) — pre-pretraining with formal languages (michahu)
• Lee et al. (2024) — NCA pre-pretraining (danihyunlee)