---
license: mit
tags:
  - educational
  - from-scratch
  - char-level
  - transformer
  - pytorch
---

# tinyshakespeare -- char tokenizer, n_embd=128

**This is an educational project, not a practical language model.** It is a
from-scratch transformer built to learn the mechanics of language modeling
end-to-end. It is not intended for any downstream or production use --
treat it as a worked example, not a tool.

## Model architecture

Token + position embeddings -> n_layer stacked transformer blocks (pre-norm multi-head causal self-attention + feedforward, residual connections) -> final LayerNorm -> linear head to vocab logits. Full hyperparameters are in `config.json`;
nothing about the architecture is hardcoded in the loader -- see
`load_model.py`.

| | |
|---|---|
| Parameters | 812,609 |
| n_embd | 128 |
| n_head | 4 |
| n_layer | 4 |
| block_size | 32 |
| Tokenizer | char-level, vocab_size=65 |

## Training corpus

Trained on **TinyShakespeare** (~1.1M characters), fetched directly from `https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt`.

## Results

Four runs, all on TinyShakespeare, same architecture family (`n_layer=4`,
`n_head=4`, `block_size=32`, `lr=1e-3`, `max_iters=8000`, `batch_size=32`,
seed=1337) — a controlled sweep over embedding width, plus one run swapping
in a trained BPE subword tokenizer instead of character-level.

**Raw cross-entropy (nats/token) is not comparable across tokenizers** —
a 1000-token BPE vocabulary and a 65-token char vocabulary have different
random-guessing floors (ln(1000) ≈ 6.9 vs ln(65) ≈ 4.2), so a higher raw
loss on BPE does not mean a worse model. **Bits-per-character (BPC)**
normalizes both onto the same unit — bits of model surprise per character
of the *original* text — and is the number to compare across the table.

| Tokenizer | n_embd | Params | Train loss (nats) | Train BPC | Val loss (nats) | Val BPC | Train/val gap (BPC) | Wall clock |
|---|---|---|---|---|---|---|---|---|
| char | 32  | 55,745  | 1.7159 | 2.4755 | 1.8849 | 2.7193 | 0.2438 | 4m48.8s |
| char | 64  | 209,729 | 1.5247 | 2.1997 | 1.7220 | 2.4843 | 0.2846 | 7m19.4s |
| char | **128** | **812,609** | **1.4043** | **2.0260** | **1.6257** | **2.3454** | 0.3194 | 16m37.6s |
| bpe (vocab=1000) | 32 | 116,520 | 3.5683 | 2.1272 | 3.8843 | 2.4189 | 0.2917 | 6m29.5s |

**Published checkpoint: char, n_embd=128** (bolded row) — lowest val BPC of
the char-level sweep.

Scaling observation: doubling `n_embd` costs a roughly constant ~3.8×
parameters each step (32→64, 64→128), but the val-BPC improvement shrinks
each time (0.235 → 0.139 in BPC terms), and the train/val gap widens —
diminishing, saturating returns from embedding width alone once depth
(`n_layer=4`) and context (`block_size=32`) are held fixed.

Tokenizer observation: char-level (BPC 2.35) still edges out this one BPE
run (BPC 2.42) at comparable parameter count, but BPE achieves that with
~7x fewer parameters than the 812K char model and visibly more coherent
generated chunks — a fair head-to-head at matched parameter count wasn't
run here.

## Files

| File | Purpose |
|---|---|
| `model.safetensors` | `state_dict()` of the trained `BigramLanguageModel`, in safetensors format |
| `config.json` | Every hyperparameter needed to reconstruct the architecture and tokenizer |
| `tokenizer_char.json` | The char-level tokenizer's vocabulary |
| `bigram.py` | Model architecture source (vendored so this folder is self-sufficient) |
| `tokenizer.py` | Tokenizer source (vendored, same reason) |
| `load_model.py` | Reconstructs the model + tokenizer from `config.json` and generates a sample |

## Usage

```bash
pip install torch safetensors
python load_model.py
```

`load_model.py` reads `config.json` for every architectural parameter --
it does not assume or hardcode them -- loads `model.safetensors`
into a freshly constructed model, and samples 400 characters to prove the
checkpoint and its config agree.

## Limitations

This is a ~65-char-vocabulary,
32-token-context toy model trained for
8,000 steps. It reproduces surface
Shakespeare-ish texture but not coherent meaning, plot, or factual
content. **Do not use this for anything beyond studying how these
mechanics fit together.**