mellogood commited on
Commit
7ce15b7
Β·
verified Β·
1 Parent(s): 4736894

docs: card follows the house-style sweep

Browse files
Files changed (1) hide show
  1. README.md +20 -20
README.md CHANGED
@@ -20,24 +20,24 @@ datasets:
20
 
21
  > A [sup computer](https://www.supcpu.com) release β€” a small language model studio. [Model page](https://www.supcpu.com/models/shakespeare-nanogpt-3/) Β· [monorepo](https://github.com/romellogoodman/sup-computer) (frozen code: [`projects/shakespeare/models/shakespeare-nanogpt-3/`](https://github.com/romellogoodman/sup-computer/tree/main/projects/shakespeare/models/shakespeare-nanogpt-3), tag `shakespeare-nanogpt-3`) Β· runs in your browser at [www.supcpu.com/model-player](https://www.supcpu.com/model-player/).
22
 
23
-
24
  <div class="takeaways">
25
  <p class="takeaways-label">Key takeaways</p>
26
  <ul>
27
- <li>The new series best: an enlarged early-modern-drama corpus + a <strong>1024-token, corpus-trained byte-level BPE</strong> + <strong>float32</strong> training, reaching held-out <code>BPC 1.831</code> at just <strong>11.02M params</strong>.</li>
28
- <li>The headline is <strong>parameter efficiency</strong>: it beats the prior champion v2 (1.919 at 29.9M) and matches-or-beats a fresh float32 GPT-2-vocab control (1.843 at 29.9M) at <strong>~1/3 the parameters</strong>.</li>
29
- <li>The BPC edge over that fresh control is only βˆ’0.012 β€” <strong>within single-seed noise</strong>. The clean wins are the params efficiency and beating the prior champion; a <strong>multi-seed</strong> run is the stated next step.</li>
30
  </ul>
31
  </div>
32
 
33
  The current best model in the [`shakespeare-nanogpt`](https://github.com/romellogoodman/sup-computer/blob/main/projects/shakespeare/README.md)
34
- series. Where [v2](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/model-cards/shakespeare-nanogpt-2.md) established the modern architecture
35
- (RoPE, RMSNorm, bias-free) on the Complete Works with GPT-2 BPE, **v3 changes the
36
- data and the tokenizer, not the architecture**: it trains a small vocabulary *on
37
- the corpus itself*, enlarges that corpus with contemporary early-modern drama, and
38
- trains in float32. The result is equal-or-better quality at about a third of the
39
- size β€” this remains **LLM-assisted research** (Claude as the *researcher* under
40
- human direction), not recursive self-improvement.
 
41
 
42
  > **Series note.** Successor to [`shakespeare-nanogpt-2`](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/model-cards/shakespeare-nanogpt-2.md).
43
  > All versions and the full story are in [`MODELS.md`](https://github.com/romellogoodman/sup-computer/blob/main/projects/shakespeare/MODELS.md);
@@ -51,8 +51,8 @@ human direction), not recursive self-improvement.
51
  | **Origin** | LLM-assisted research, round 6 float32 re-run (`projects/shakespeare/runs/r6-fp32-bpe1k`) |
52
  | **Architecture** | modern β€” RoPE, RMSNorm, bias-free (`core/nanogpt_core/model.py`, vendored into the frozen folder) |
53
  | **Size** | ~11.02M params (6 layers Β· 6 heads Β· 384 embd Β· 256 context) |
54
- | **Tokenizer** | **1024-vocab byte-level BPE**, trained on the enlarged corpus (committed `tokenizer.json`; the `meta.pkl` seam, ADR-0012) |
55
- | **Precision** | **float32** (eliminates the MPS float16 large-vocab logit overflow that confounded round 5) |
56
  | **Checkpoint** | `models/shakespeare-nanogpt-3/ckpt.pt` (weights not committed β€” rebuild below) |
57
  | **Built on** | [nanoGPT](https://github.com/karpathy/nanoGPT) by Andrej Karpathy (MIT) |
58
  | **Developed with** | Claude Fable 5 ([Claude Code](https://claude.com/claude-code)) as researcher, human oversight |
@@ -68,7 +68,7 @@ stage-direction convention of the Marlowe/Webster editions.
68
 
69
  **Out of scope:** real use of the text; any presentation of output as genuine
70
  Shakespeare (or Marlowe, Jonson, etc.) or as fact. No instruction following, no
71
- safety tuning. This is **mimicry only**.
72
 
73
  ## Training data
74
 
@@ -77,14 +77,14 @@ The **enlarged early-modern-drama corpus**: Shakespeare's Complete Works
77
  Faustus*, both *Tamburlaine*s, *Edward II*, *The Jew of Malta*), Jonson
78
  (*Volpone*, *The Alchemist*, *Every Man in His Humour*), Kyd (*The Spanish
79
  Tragedy*), Webster (*The Duchess of Malfi*, *The White Devil*), and Dekker. Total
80
- training text ~7.85M characters; the tokenizer is trained on the **training split
81
- only**.
82
 
83
  Crucially, the **held-out test set is unchanged**: the same fixed
84
  250k-character Shakespeare slice (`projects/shakespeare/test.txt`) every version
85
  in the series is scored on. It is excluded from training and never duplicated β€”
86
  so v3's BPC stays directly comparable to v1 and v2 despite the larger, broader
87
- training corpus. Enlarging the corpus also **eliminated the overfit** that defined
88
  v2's rounds: validation loss fell monotonically instead of bottoming early.
89
 
90
  ## Training procedure
@@ -115,7 +115,7 @@ comparable. Lower is better.
115
  Two clean results and one honest caveat:
116
 
117
  - **Parameter efficiency (clean win).** v3 reaches equal-or-better BPC than the
118
- 50k-vocab models at **~1/3 the parameters**. Of v2's 29.9M parameters, ~19.3M
119
  *was* the GPT-2 embedding table; a 1024-vocab embedding is ~0.4M. That budget
120
  buys capability instead of a giant vocabulary the Shakespeare domain never uses.
121
  - **Beats the prior champion (clean win).** 1.831 < v2's 1.919 (βˆ’4.6%). All three
@@ -142,7 +142,7 @@ Two clean results and one honest caveat:
142
  (`1337`), no variance estimate. v3's win over the *fresh float32 control* and its
143
  gap to the unreleased `bpe4k` are within plausible run-to-run noise. The clear
144
  results (params efficiency; beating the prior champion) do not depend on that
145
- narrow margin. **Multi-seed replication is the explicit next step** β€” it is what
146
  would turn "bpe1k is tied-best" into "bpe1k is best."
147
  - **`bpe1k` was still improving.** In round 5 its validation loss had not
148
  plateaued at 2000 iterations; more iterations would likely lower BPC further, so
@@ -166,7 +166,7 @@ uv run --with tokenizers python eval.py # score on the shared held-out tes
166
  uv run --with tokenizers python sample.py --start="ROMEO:"
167
  ```
168
 
169
- The 1024-token `tokenizer.json` is committed and **never retrained** β€” `prepare.py`
170
  only re-encodes with it, pinning the exact vocabulary.
171
 
172
  ## Citation / credits
 
20
 
21
  > A [sup computer](https://www.supcpu.com) release β€” a small language model studio. [Model page](https://www.supcpu.com/models/shakespeare-nanogpt-3/) Β· [monorepo](https://github.com/romellogoodman/sup-computer) (frozen code: [`projects/shakespeare/models/shakespeare-nanogpt-3/`](https://github.com/romellogoodman/sup-computer/tree/main/projects/shakespeare/models/shakespeare-nanogpt-3), tag `shakespeare-nanogpt-3`) Β· runs in your browser at [www.supcpu.com/model-player](https://www.supcpu.com/model-player/).
22
 
 
23
  <div class="takeaways">
24
  <p class="takeaways-label">Key takeaways</p>
25
  <ul>
26
+ <li>The new series best: an enlarged early-modern-drama corpus + a <strong>1024-token, corpus-trained byte-level BPE</strong> + float32 training, reaching held-out <code>BPC 1.831</code> at just 11.02M params.</li>
27
+ <li>The headline is <strong>parameter efficiency</strong>: it beats the prior champion v2 (1.919 at 29.9M) and matches-or-beats a fresh float32 GPT-2-vocab control (1.843 at 29.9M) at ~1/3 the parameters.</li>
28
+ <li>The BPC edge over that fresh control is only βˆ’0.012 β€” <strong>within single-seed noise</strong>. The clean wins are the params efficiency and beating the prior champion; a multi-seed run is the stated next step.</li>
29
  </ul>
30
  </div>
31
 
32
  The current best model in the [`shakespeare-nanogpt`](https://github.com/romellogoodman/sup-computer/blob/main/projects/shakespeare/README.md)
33
+ series: held-out BPC 1.831 at ~11.02M params, about a third the size of the
34
+ champion it beats. Where [v2](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/model-cards/shakespeare-nanogpt-2.md) established the modern
35
+ architecture (RoPE, RMSNorm, bias-free) on the Complete Works with GPT-2 BPE,
36
+ **v3 changes the data and the tokenizer, not the architecture**: it trains a
37
+ small vocabulary *on the corpus itself*, enlarges that corpus with contemporary
38
+ early-modern drama, and trains in float32. This remains LLM-assisted research β€”
39
+ Claude as the *researcher* under human direction β€” not recursive
40
+ self-improvement.
41
 
42
  > **Series note.** Successor to [`shakespeare-nanogpt-2`](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/model-cards/shakespeare-nanogpt-2.md).
43
  > All versions and the full story are in [`MODELS.md`](https://github.com/romellogoodman/sup-computer/blob/main/projects/shakespeare/MODELS.md);
 
51
  | **Origin** | LLM-assisted research, round 6 float32 re-run (`projects/shakespeare/runs/r6-fp32-bpe1k`) |
52
  | **Architecture** | modern β€” RoPE, RMSNorm, bias-free (`core/nanogpt_core/model.py`, vendored into the frozen folder) |
53
  | **Size** | ~11.02M params (6 layers Β· 6 heads Β· 384 embd Β· 256 context) |
54
+ | **Tokenizer** | 1024-vocab byte-level BPE, trained on the enlarged corpus (committed `tokenizer.json`; the `meta.pkl` seam, ADR-0012) |
55
+ | **Precision** | float32 (eliminates the MPS float16 large-vocab logit overflow that confounded round 5) |
56
  | **Checkpoint** | `models/shakespeare-nanogpt-3/ckpt.pt` (weights not committed β€” rebuild below) |
57
  | **Built on** | [nanoGPT](https://github.com/karpathy/nanoGPT) by Andrej Karpathy (MIT) |
58
  | **Developed with** | Claude Fable 5 ([Claude Code](https://claude.com/claude-code)) as researcher, human oversight |
 
68
 
69
  **Out of scope:** real use of the text; any presentation of output as genuine
70
  Shakespeare (or Marlowe, Jonson, etc.) or as fact. No instruction following, no
71
+ safety tuning. This is mimicry only.
72
 
73
  ## Training data
74
 
 
77
  Faustus*, both *Tamburlaine*s, *Edward II*, *The Jew of Malta*), Jonson
78
  (*Volpone*, *The Alchemist*, *Every Man in His Humour*), Kyd (*The Spanish
79
  Tragedy*), Webster (*The Duchess of Malfi*, *The White Devil*), and Dekker. Total
80
+ training text ~7.85M characters; the tokenizer is trained on the training split
81
+ only.
82
 
83
  Crucially, the **held-out test set is unchanged**: the same fixed
84
  250k-character Shakespeare slice (`projects/shakespeare/test.txt`) every version
85
  in the series is scored on. It is excluded from training and never duplicated β€”
86
  so v3's BPC stays directly comparable to v1 and v2 despite the larger, broader
87
+ training corpus. Enlarging the corpus also eliminated the overfit that defined
88
  v2's rounds: validation loss fell monotonically instead of bottoming early.
89
 
90
  ## Training procedure
 
115
  Two clean results and one honest caveat:
116
 
117
  - **Parameter efficiency (clean win).** v3 reaches equal-or-better BPC than the
118
+ 50k-vocab models at ~1/3 the parameters. Of v2's 29.9M parameters, ~19.3M
119
  *was* the GPT-2 embedding table; a 1024-vocab embedding is ~0.4M. That budget
120
  buys capability instead of a giant vocabulary the Shakespeare domain never uses.
121
  - **Beats the prior champion (clean win).** 1.831 < v2's 1.919 (βˆ’4.6%). All three
 
142
  (`1337`), no variance estimate. v3's win over the *fresh float32 control* and its
143
  gap to the unreleased `bpe4k` are within plausible run-to-run noise. The clear
144
  results (params efficiency; beating the prior champion) do not depend on that
145
+ narrow margin. Multi-seed replication is the explicit next step β€” it is what
146
  would turn "bpe1k is tied-best" into "bpe1k is best."
147
  - **`bpe1k` was still improving.** In round 5 its validation loss had not
148
  plateaued at 2000 iterations; more iterations would likely lower BPC further, so
 
166
  uv run --with tokenizers python sample.py --start="ROMEO:"
167
  ```
168
 
169
+ The 1024-token `tokenizer.json` is committed and never retrained β€” `prepare.py`
170
  only re-encodes with it, pinning the exact vocabulary.
171
 
172
  ## Citation / credits