---
license: apache-2.0
datasets:
- trixyL/simplestories-8k-megatron
language:
- en
pipeline_tag: text-generation
tags:
- simplestories
- '512'
- 8k
- transformerlm
---
# 🧠✨ TransformerLM (AR 512, 8K vocab) — SimpleStories

This is the result of the code from https://github.com/triloy8/transformerlm, a minimal autoregressive Transformer LM trained on **SimpleStories** with a **512-token context** and an **8K vocab tokenizer**. ✨

## ✅ Key Facts

- **Model type:** Autoregressive Transformer LM
- **Dataset:** SimpleStories
- **Context length:** 512 tokens
- **Tokenizer vocab size:** 8,000
- **Layers:** 12
- **Heads:** 8
- **d_model:** 512
- **d_ff:** 2,048
- **Training setup:** Single **NVIDIA A40 48GB**
- **Runtime:** ~20 hours ⏱️

## 📦 What’s Inside

- 60k steps from a 60k run, including:
  - Optimizer state
  - RNG state
  - Safetensors weights
- Tokenizer config
- Run config

## 🚀 Reproducibility

To reproduce the run:

Exact commit that launched the train: https://github.com/triloy8/transformerlm/commit/06cb4831d47c04a18573bee8e28dc83b10086d06