Pythia-160M Pre-Pretraining: music_steps500 (seed 208)
Trained from scratch using the ppt pre-pretraining research framework.
Training Details
| Parameter | Value |
|---|---|
| Base architecture | EleutherAI/pythia-160m (reinitialized) |
| Regimen | music_steps500 |
| Seed | 208 |
| Stage 1 dataset | MAESTRO MIDI music tokens |
| Stage 1 steps | 500 |
| Stage 2 dataset | OpenWebText |
| Stage 2 steps | 10000 |
| Optimizer | AdamW (lr=1e-3, wd=0.0) |
| Effective batch size | 64 |
| Sequence length | 2048 |
Control Design
Stage 1: MIDI music event tokens from MAESTRO dataset. Musical structure (rhythm, harmony) provides hierarchical patterns.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("sashaboguraev/pythia-160m-ppt-music_steps500-seed208")
tokenizer = AutoTokenizer.from_pretrained("sashaboguraev/pythia-160m-ppt-music_steps500-seed208")
Citation
If you use this model, please cite the original pre-pretraining papers:
- Papadimitriou & Jurafsky (2020) — tilt-transfer
- Hahn & Rofin (2024) — pre-pretraining with formal languages (michahu)
- Lee et al. (2024) — NCA pre-pretraining (danihyunlee)
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for sashaboguraev/pythia-160m-ppt-music_steps500-seed208
Base model
EleutherAI/pythia-160m