docs: card follows the house-style sweep
Browse files
README.md
CHANGED
|
@@ -20,24 +20,24 @@ datasets:
|
|
| 20 |
|
| 21 |
> A [sup computer](https://www.supcpu.com) release β a small language model studio. [Model page](https://www.supcpu.com/models/shakespeare-nanogpt-3/) Β· [monorepo](https://github.com/romellogoodman/sup-computer) (frozen code: [`projects/shakespeare/models/shakespeare-nanogpt-3/`](https://github.com/romellogoodman/sup-computer/tree/main/projects/shakespeare/models/shakespeare-nanogpt-3), tag `shakespeare-nanogpt-3`) Β· runs in your browser at [www.supcpu.com/model-player](https://www.supcpu.com/model-player/).
|
| 22 |
|
| 23 |
-
|
| 24 |
<div class="takeaways">
|
| 25 |
<p class="takeaways-label">Key takeaways</p>
|
| 26 |
<ul>
|
| 27 |
-
<li>The new series best: an enlarged early-modern-drama corpus + a <strong>1024-token, corpus-trained byte-level BPE</strong> +
|
| 28 |
-
<li>The headline is <strong>parameter efficiency</strong>: it beats the prior champion v2 (1.919 at 29.9M) and matches-or-beats a fresh float32 GPT-2-vocab control (1.843 at 29.9M) at
|
| 29 |
-
<li>The BPC edge over that fresh control is only β0.012 β <strong>within single-seed noise</strong>. The clean wins are the params efficiency and beating the prior champion; a
|
| 30 |
</ul>
|
| 31 |
</div>
|
| 32 |
|
| 33 |
The current best model in the [`shakespeare-nanogpt`](https://github.com/romellogoodman/sup-computer/blob/main/projects/shakespeare/README.md)
|
| 34 |
-
series.
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
the
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
human direction
|
|
|
|
| 41 |
|
| 42 |
> **Series note.** Successor to [`shakespeare-nanogpt-2`](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/model-cards/shakespeare-nanogpt-2.md).
|
| 43 |
> All versions and the full story are in [`MODELS.md`](https://github.com/romellogoodman/sup-computer/blob/main/projects/shakespeare/MODELS.md);
|
|
@@ -51,8 +51,8 @@ human direction), not recursive self-improvement.
|
|
| 51 |
| **Origin** | LLM-assisted research, round 6 float32 re-run (`projects/shakespeare/runs/r6-fp32-bpe1k`) |
|
| 52 |
| **Architecture** | modern β RoPE, RMSNorm, bias-free (`core/nanogpt_core/model.py`, vendored into the frozen folder) |
|
| 53 |
| **Size** | ~11.02M params (6 layers Β· 6 heads Β· 384 embd Β· 256 context) |
|
| 54 |
-
| **Tokenizer** |
|
| 55 |
-
| **Precision** |
|
| 56 |
| **Checkpoint** | `models/shakespeare-nanogpt-3/ckpt.pt` (weights not committed β rebuild below) |
|
| 57 |
| **Built on** | [nanoGPT](https://github.com/karpathy/nanoGPT) by Andrej Karpathy (MIT) |
|
| 58 |
| **Developed with** | Claude Fable 5 ([Claude Code](https://claude.com/claude-code)) as researcher, human oversight |
|
|
@@ -68,7 +68,7 @@ stage-direction convention of the Marlowe/Webster editions.
|
|
| 68 |
|
| 69 |
**Out of scope:** real use of the text; any presentation of output as genuine
|
| 70 |
Shakespeare (or Marlowe, Jonson, etc.) or as fact. No instruction following, no
|
| 71 |
-
safety tuning. This is
|
| 72 |
|
| 73 |
## Training data
|
| 74 |
|
|
@@ -77,14 +77,14 @@ The **enlarged early-modern-drama corpus**: Shakespeare's Complete Works
|
|
| 77 |
Faustus*, both *Tamburlaine*s, *Edward II*, *The Jew of Malta*), Jonson
|
| 78 |
(*Volpone*, *The Alchemist*, *Every Man in His Humour*), Kyd (*The Spanish
|
| 79 |
Tragedy*), Webster (*The Duchess of Malfi*, *The White Devil*), and Dekker. Total
|
| 80 |
-
training text ~7.85M characters; the tokenizer is trained on the
|
| 81 |
-
only
|
| 82 |
|
| 83 |
Crucially, the **held-out test set is unchanged**: the same fixed
|
| 84 |
250k-character Shakespeare slice (`projects/shakespeare/test.txt`) every version
|
| 85 |
in the series is scored on. It is excluded from training and never duplicated β
|
| 86 |
so v3's BPC stays directly comparable to v1 and v2 despite the larger, broader
|
| 87 |
-
training corpus. Enlarging the corpus also
|
| 88 |
v2's rounds: validation loss fell monotonically instead of bottoming early.
|
| 89 |
|
| 90 |
## Training procedure
|
|
@@ -115,7 +115,7 @@ comparable. Lower is better.
|
|
| 115 |
Two clean results and one honest caveat:
|
| 116 |
|
| 117 |
- **Parameter efficiency (clean win).** v3 reaches equal-or-better BPC than the
|
| 118 |
-
50k-vocab models at
|
| 119 |
*was* the GPT-2 embedding table; a 1024-vocab embedding is ~0.4M. That budget
|
| 120 |
buys capability instead of a giant vocabulary the Shakespeare domain never uses.
|
| 121 |
- **Beats the prior champion (clean win).** 1.831 < v2's 1.919 (β4.6%). All three
|
|
@@ -142,7 +142,7 @@ Two clean results and one honest caveat:
|
|
| 142 |
(`1337`), no variance estimate. v3's win over the *fresh float32 control* and its
|
| 143 |
gap to the unreleased `bpe4k` are within plausible run-to-run noise. The clear
|
| 144 |
results (params efficiency; beating the prior champion) do not depend on that
|
| 145 |
-
narrow margin.
|
| 146 |
would turn "bpe1k is tied-best" into "bpe1k is best."
|
| 147 |
- **`bpe1k` was still improving.** In round 5 its validation loss had not
|
| 148 |
plateaued at 2000 iterations; more iterations would likely lower BPC further, so
|
|
@@ -166,7 +166,7 @@ uv run --with tokenizers python eval.py # score on the shared held-out tes
|
|
| 166 |
uv run --with tokenizers python sample.py --start="ROMEO:"
|
| 167 |
```
|
| 168 |
|
| 169 |
-
The 1024-token `tokenizer.json` is committed and
|
| 170 |
only re-encodes with it, pinning the exact vocabulary.
|
| 171 |
|
| 172 |
## Citation / credits
|
|
|
|
| 20 |
|
| 21 |
> A [sup computer](https://www.supcpu.com) release β a small language model studio. [Model page](https://www.supcpu.com/models/shakespeare-nanogpt-3/) Β· [monorepo](https://github.com/romellogoodman/sup-computer) (frozen code: [`projects/shakespeare/models/shakespeare-nanogpt-3/`](https://github.com/romellogoodman/sup-computer/tree/main/projects/shakespeare/models/shakespeare-nanogpt-3), tag `shakespeare-nanogpt-3`) Β· runs in your browser at [www.supcpu.com/model-player](https://www.supcpu.com/model-player/).
|
| 22 |
|
|
|
|
| 23 |
<div class="takeaways">
|
| 24 |
<p class="takeaways-label">Key takeaways</p>
|
| 25 |
<ul>
|
| 26 |
+
<li>The new series best: an enlarged early-modern-drama corpus + a <strong>1024-token, corpus-trained byte-level BPE</strong> + float32 training, reaching held-out <code>BPC 1.831</code> at just 11.02M params.</li>
|
| 27 |
+
<li>The headline is <strong>parameter efficiency</strong>: it beats the prior champion v2 (1.919 at 29.9M) and matches-or-beats a fresh float32 GPT-2-vocab control (1.843 at 29.9M) at ~1/3 the parameters.</li>
|
| 28 |
+
<li>The BPC edge over that fresh control is only β0.012 β <strong>within single-seed noise</strong>. The clean wins are the params efficiency and beating the prior champion; a multi-seed run is the stated next step.</li>
|
| 29 |
</ul>
|
| 30 |
</div>
|
| 31 |
|
| 32 |
The current best model in the [`shakespeare-nanogpt`](https://github.com/romellogoodman/sup-computer/blob/main/projects/shakespeare/README.md)
|
| 33 |
+
series: held-out BPC 1.831 at ~11.02M params, about a third the size of the
|
| 34 |
+
champion it beats. Where [v2](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/model-cards/shakespeare-nanogpt-2.md) established the modern
|
| 35 |
+
architecture (RoPE, RMSNorm, bias-free) on the Complete Works with GPT-2 BPE,
|
| 36 |
+
**v3 changes the data and the tokenizer, not the architecture**: it trains a
|
| 37 |
+
small vocabulary *on the corpus itself*, enlarges that corpus with contemporary
|
| 38 |
+
early-modern drama, and trains in float32. This remains LLM-assisted research β
|
| 39 |
+
Claude as the *researcher* under human direction β not recursive
|
| 40 |
+
self-improvement.
|
| 41 |
|
| 42 |
> **Series note.** Successor to [`shakespeare-nanogpt-2`](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/model-cards/shakespeare-nanogpt-2.md).
|
| 43 |
> All versions and the full story are in [`MODELS.md`](https://github.com/romellogoodman/sup-computer/blob/main/projects/shakespeare/MODELS.md);
|
|
|
|
| 51 |
| **Origin** | LLM-assisted research, round 6 float32 re-run (`projects/shakespeare/runs/r6-fp32-bpe1k`) |
|
| 52 |
| **Architecture** | modern β RoPE, RMSNorm, bias-free (`core/nanogpt_core/model.py`, vendored into the frozen folder) |
|
| 53 |
| **Size** | ~11.02M params (6 layers Β· 6 heads Β· 384 embd Β· 256 context) |
|
| 54 |
+
| **Tokenizer** | 1024-vocab byte-level BPE, trained on the enlarged corpus (committed `tokenizer.json`; the `meta.pkl` seam, ADR-0012) |
|
| 55 |
+
| **Precision** | float32 (eliminates the MPS float16 large-vocab logit overflow that confounded round 5) |
|
| 56 |
| **Checkpoint** | `models/shakespeare-nanogpt-3/ckpt.pt` (weights not committed β rebuild below) |
|
| 57 |
| **Built on** | [nanoGPT](https://github.com/karpathy/nanoGPT) by Andrej Karpathy (MIT) |
|
| 58 |
| **Developed with** | Claude Fable 5 ([Claude Code](https://claude.com/claude-code)) as researcher, human oversight |
|
|
|
|
| 68 |
|
| 69 |
**Out of scope:** real use of the text; any presentation of output as genuine
|
| 70 |
Shakespeare (or Marlowe, Jonson, etc.) or as fact. No instruction following, no
|
| 71 |
+
safety tuning. This is mimicry only.
|
| 72 |
|
| 73 |
## Training data
|
| 74 |
|
|
|
|
| 77 |
Faustus*, both *Tamburlaine*s, *Edward II*, *The Jew of Malta*), Jonson
|
| 78 |
(*Volpone*, *The Alchemist*, *Every Man in His Humour*), Kyd (*The Spanish
|
| 79 |
Tragedy*), Webster (*The Duchess of Malfi*, *The White Devil*), and Dekker. Total
|
| 80 |
+
training text ~7.85M characters; the tokenizer is trained on the training split
|
| 81 |
+
only.
|
| 82 |
|
| 83 |
Crucially, the **held-out test set is unchanged**: the same fixed
|
| 84 |
250k-character Shakespeare slice (`projects/shakespeare/test.txt`) every version
|
| 85 |
in the series is scored on. It is excluded from training and never duplicated β
|
| 86 |
so v3's BPC stays directly comparable to v1 and v2 despite the larger, broader
|
| 87 |
+
training corpus. Enlarging the corpus also eliminated the overfit that defined
|
| 88 |
v2's rounds: validation loss fell monotonically instead of bottoming early.
|
| 89 |
|
| 90 |
## Training procedure
|
|
|
|
| 115 |
Two clean results and one honest caveat:
|
| 116 |
|
| 117 |
- **Parameter efficiency (clean win).** v3 reaches equal-or-better BPC than the
|
| 118 |
+
50k-vocab models at ~1/3 the parameters. Of v2's 29.9M parameters, ~19.3M
|
| 119 |
*was* the GPT-2 embedding table; a 1024-vocab embedding is ~0.4M. That budget
|
| 120 |
buys capability instead of a giant vocabulary the Shakespeare domain never uses.
|
| 121 |
- **Beats the prior champion (clean win).** 1.831 < v2's 1.919 (β4.6%). All three
|
|
|
|
| 142 |
(`1337`), no variance estimate. v3's win over the *fresh float32 control* and its
|
| 143 |
gap to the unreleased `bpe4k` are within plausible run-to-run noise. The clear
|
| 144 |
results (params efficiency; beating the prior champion) do not depend on that
|
| 145 |
+
narrow margin. Multi-seed replication is the explicit next step β it is what
|
| 146 |
would turn "bpe1k is tied-best" into "bpe1k is best."
|
| 147 |
- **`bpe1k` was still improving.** In round 5 its validation loss had not
|
| 148 |
plateaued at 2000 iterations; more iterations would likely lower BPC further, so
|
|
|
|
| 166 |
uv run --with tokenizers python sample.py --start="ROMEO:"
|
| 167 |
```
|
| 168 |
|
| 169 |
+
The 1024-token `tokenizer.json` is committed and never retrained β `prepare.py`
|
| 170 |
only re-encodes with it, pinning the exact vocabulary.
|
| 171 |
|
| 172 |
## Citation / credits
|