File size: 4,287 Bytes
1b170a9 436c9f7 50aa7f9 436c9f7 1b170a9 436c9f7 50aa7f9 436c9f7 50aa7f9 436c9f7 50aa7f9 436c9f7 50aa7f9 436c9f7 50aa7f9 436c9f7 50aa7f9 436c9f7 50aa7f9 436c9f7 50aa7f9 436c9f7 50aa7f9 436c9f7 50aa7f9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 | ---
license: mit
tags:
- educational
- from-scratch
- char-level
- transformer
- pytorch
---
# tinyshakespeare -- char tokenizer, n_embd=128
**This is an educational project, not a practical language model.** It is a
from-scratch transformer built to learn the mechanics of language modeling
end-to-end. It is not intended for any downstream or production use --
treat it as a worked example, not a tool.
## Model architecture
Token + position embeddings -> n_layer stacked transformer blocks (pre-norm multi-head causal self-attention + feedforward, residual connections) -> final LayerNorm -> linear head to vocab logits. Full hyperparameters are in `config.json`;
nothing about the architecture is hardcoded in the loader -- see
`load_model.py`.
| | |
|---|---|
| Parameters | 812,609 |
| n_embd | 128 |
| n_head | 4 |
| n_layer | 4 |
| block_size | 32 |
| Tokenizer | char-level, vocab_size=65 |
## Training corpus
Trained on **TinyShakespeare** (~1.1M characters), fetched directly from `https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt`.
## Results
Four runs, all on TinyShakespeare, same architecture family (`n_layer=4`,
`n_head=4`, `block_size=32`, `lr=1e-3`, `max_iters=8000`, `batch_size=32`,
seed=1337) — a controlled sweep over embedding width, plus one run swapping
in a trained BPE subword tokenizer instead of character-level.
**Raw cross-entropy (nats/token) is not comparable across tokenizers** —
a 1000-token BPE vocabulary and a 65-token char vocabulary have different
random-guessing floors (ln(1000) ≈ 6.9 vs ln(65) ≈ 4.2), so a higher raw
loss on BPE does not mean a worse model. **Bits-per-character (BPC)**
normalizes both onto the same unit — bits of model surprise per character
of the *original* text — and is the number to compare across the table.
| Tokenizer | n_embd | Params | Train loss (nats) | Train BPC | Val loss (nats) | Val BPC | Train/val gap (BPC) | Wall clock |
|---|---|---|---|---|---|---|---|---|
| char | 32 | 55,745 | 1.7159 | 2.4755 | 1.8849 | 2.7193 | 0.2438 | 4m48.8s |
| char | 64 | 209,729 | 1.5247 | 2.1997 | 1.7220 | 2.4843 | 0.2846 | 7m19.4s |
| char | **128** | **812,609** | **1.4043** | **2.0260** | **1.6257** | **2.3454** | 0.3194 | 16m37.6s |
| bpe (vocab=1000) | 32 | 116,520 | 3.5683 | 2.1272 | 3.8843 | 2.4189 | 0.2917 | 6m29.5s |
**Published checkpoint: char, n_embd=128** (bolded row) — lowest val BPC of
the char-level sweep.
Scaling observation: doubling `n_embd` costs a roughly constant ~3.8×
parameters each step (32→64, 64→128), but the val-BPC improvement shrinks
each time (0.235 → 0.139 in BPC terms), and the train/val gap widens —
diminishing, saturating returns from embedding width alone once depth
(`n_layer=4`) and context (`block_size=32`) are held fixed.
Tokenizer observation: char-level (BPC 2.35) still edges out this one BPE
run (BPC 2.42) at comparable parameter count, but BPE achieves that with
~7x fewer parameters than the 812K char model and visibly more coherent
generated chunks — a fair head-to-head at matched parameter count wasn't
run here.
## Files
| File | Purpose |
|---|---|
| `model.safetensors` | `state_dict()` of the trained `BigramLanguageModel`, in safetensors format |
| `config.json` | Every hyperparameter needed to reconstruct the architecture and tokenizer |
| `tokenizer_char.json` | The char-level tokenizer's vocabulary |
| `bigram.py` | Model architecture source (vendored so this folder is self-sufficient) |
| `tokenizer.py` | Tokenizer source (vendored, same reason) |
| `load_model.py` | Reconstructs the model + tokenizer from `config.json` and generates a sample |
## Usage
```bash
pip install torch safetensors
python load_model.py
```
`load_model.py` reads `config.json` for every architectural parameter --
it does not assume or hardcode them -- loads `model.safetensors`
into a freshly constructed model, and samples 400 characters to prove the
checkpoint and its config agree.
## Limitations
This is a ~65-char-vocabulary,
32-token-context toy model trained for
8,000 steps. It reproduces surface
Shakespeare-ish texture but not coherent meaning, plot, or factual
content. **Do not use this for anything beyond studying how these
mechanics fit together.**
|