| --- |
| license: mit |
| tags: |
| - educational |
| - from-scratch |
| - char-level |
| - transformer |
| - pytorch |
| --- |
| |
| # tinyshakespeare -- char tokenizer, n_embd=128 |
| |
| **This is an educational project, not a practical language model.** It is a |
| from-scratch transformer built to learn the mechanics of language modeling |
| end-to-end. It is not intended for any downstream or production use -- |
| treat it as a worked example, not a tool. |
| |
| ## Model architecture |
| |
| Token + position embeddings -> n_layer stacked transformer blocks (pre-norm multi-head causal self-attention + feedforward, residual connections) -> final LayerNorm -> linear head to vocab logits. Full hyperparameters are in `config.json`; |
| nothing about the architecture is hardcoded in the loader -- see |
| `load_model.py`. |
|
|
| | | | |
| |---|---| |
| | Parameters | 812,609 | |
| | n_embd | 128 | |
| | n_head | 4 | |
| | n_layer | 4 | |
| | block_size | 32 | |
| | Tokenizer | char-level, vocab_size=65 | |
| |
| ## Training corpus |
| |
| Trained on **TinyShakespeare** (~1.1M characters), fetched directly from `https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt`. |
| |
| ## Results |
| |
| Four runs, all on TinyShakespeare, same architecture family (`n_layer=4`, |
| `n_head=4`, `block_size=32`, `lr=1e-3`, `max_iters=8000`, `batch_size=32`, |
| seed=1337) — a controlled sweep over embedding width, plus one run swapping |
| in a trained BPE subword tokenizer instead of character-level. |
|
|
| **Raw cross-entropy (nats/token) is not comparable across tokenizers** — |
| a 1000-token BPE vocabulary and a 65-token char vocabulary have different |
| random-guessing floors (ln(1000) ≈ 6.9 vs ln(65) ≈ 4.2), so a higher raw |
| loss on BPE does not mean a worse model. **Bits-per-character (BPC)** |
| normalizes both onto the same unit — bits of model surprise per character |
| of the *original* text — and is the number to compare across the table. |
|
|
| | Tokenizer | n_embd | Params | Train loss (nats) | Train BPC | Val loss (nats) | Val BPC | Train/val gap (BPC) | Wall clock | |
| |---|---|---|---|---|---|---|---|---| |
| | char | 32 | 55,745 | 1.7159 | 2.4755 | 1.8849 | 2.7193 | 0.2438 | 4m48.8s | |
| | char | 64 | 209,729 | 1.5247 | 2.1997 | 1.7220 | 2.4843 | 0.2846 | 7m19.4s | |
| | char | **128** | **812,609** | **1.4043** | **2.0260** | **1.6257** | **2.3454** | 0.3194 | 16m37.6s | |
| | bpe (vocab=1000) | 32 | 116,520 | 3.5683 | 2.1272 | 3.8843 | 2.4189 | 0.2917 | 6m29.5s | |
| |
| **Published checkpoint: char, n_embd=128** (bolded row) — lowest val BPC of |
| the char-level sweep. |
| |
| Scaling observation: doubling `n_embd` costs a roughly constant ~3.8× |
| parameters each step (32→64, 64→128), but the val-BPC improvement shrinks |
| each time (0.235 → 0.139 in BPC terms), and the train/val gap widens — |
| diminishing, saturating returns from embedding width alone once depth |
| (`n_layer=4`) and context (`block_size=32`) are held fixed. |
|
|
| Tokenizer observation: char-level (BPC 2.35) still edges out this one BPE |
| run (BPC 2.42) at comparable parameter count, but BPE achieves that with |
| ~7x fewer parameters than the 812K char model and visibly more coherent |
| generated chunks — a fair head-to-head at matched parameter count wasn't |
| run here. |
|
|
| ## Files |
|
|
| | File | Purpose | |
| |---|---| |
| | `model.safetensors` | `state_dict()` of the trained `BigramLanguageModel`, in safetensors format | |
| | `config.json` | Every hyperparameter needed to reconstruct the architecture and tokenizer | |
| | `tokenizer_char.json` | The char-level tokenizer's vocabulary | |
| | `bigram.py` | Model architecture source (vendored so this folder is self-sufficient) | |
| | `tokenizer.py` | Tokenizer source (vendored, same reason) | |
| | `load_model.py` | Reconstructs the model + tokenizer from `config.json` and generates a sample | |
|
|
| ## Usage |
|
|
| ```bash |
| pip install torch safetensors |
| python load_model.py |
| ``` |
|
|
| `load_model.py` reads `config.json` for every architectural parameter -- |
| it does not assume or hardcode them -- loads `model.safetensors` |
| into a freshly constructed model, and samples 400 characters to prove the |
| checkpoint and its config agree. |
|
|
| ## Limitations |
|
|
| This is a ~65-char-vocabulary, |
| 32-token-context toy model trained for |
| 8,000 steps. It reproduces surface |
| Shakespeare-ish texture but not coherent meaning, plot, or factual |
| content. **Do not use this for anything beyond studying how these |
| mechanics fit together.** |
|
|
