mellogood commited on
Commit
1f6acdd
Β·
verified Β·
1 Parent(s): e55784a

docs: card follows the house-style sweep

Browse files
Files changed (1) hide show
  1. README.md +28 -28
README.md CHANGED
@@ -18,22 +18,22 @@ tags:
18
 
19
  > A [sup computer](https://www.supcpu.com) release β€” a small language model studio. [Model page](https://www.supcpu.com/models/gatsby-nanogpt-2/) Β· [monorepo](https://github.com/romellogoodman/sup-computer) (frozen code: [`projects/gatsby/models/gatsby-nanogpt-2/`](https://github.com/romellogoodman/sup-computer/tree/main/projects/gatsby/models/gatsby-nanogpt-2), tag `gatsby-nanogpt-2`) Β· runs in your browser at [www.supcpu.com/model-player](https://www.supcpu.com/model-player/).
20
 
21
-
22
  <div class="takeaways">
23
  <p class="takeaways-label">Key takeaways</p>
24
  <ul>
25
- <li>A char-level GPT behaviourally <strong>peer to the paid baseline</strong> (<code>gatsby-nanogpt-1</code>) β€” the same green-light obsession and working <code>green=1..5</code> dial β€” but its corpus was written by a <strong>mixture of four local open models</strong> (Olmo, Ministral, Gemma, Granite) for <strong>$0</strong> instead of ~$6 of Claude API.</li>
26
- <li>The headline finding is about <strong>the blend, not the pipeline</strong>: a Granite-heavy first round broke the dial flat, because Granite barely modulates the green light across levels. <strong>Which generators you lean on is a design decision with teeth.</strong></li>
27
  <li>Rebalancing off Granite and <strong>doubling the corpus</strong> (1kβ†’2k stories) recovered the dial β€” the model needed the extra headroom to learn the conditioning the corpus already contained.</li>
28
  <li>Same status as v1: a documented <strong>milestone, not exhibit-ready</strong>. Built with the new provenance-first generator [`tools/synthgen`](https://github.com/romellogoodman/sup-computer/blob/main/tools/synthgen/README.md) ([ADR-0014](https://github.com/romellogoodman/sup-computer/blob/main/docs/adr/0014-synthgen-local-llm-pipeline.md)).</li>
29
  </ul>
30
  </div>
31
 
32
- A character-level GPT fixated on **Jay Gatsby's green light**, with a baked-in
33
- **intensity dial** (`[green=1]` undertow β†’ `[green=5]` swallows the story) β€” the
34
- same behaviour as [`gatsby-nanogpt-1`](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/model-cards/gatsby-nanogpt-1.md), but trained on a
35
- corpus written by a **mixture of four local open models** (Olmo 3, Ministral 3,
36
- Gemma 4, Granite 4.1) instead of the Claude API. Cost to write the corpus: **$0**.
 
37
  Second model in the [`gatsby-nanogpt`](https://github.com/romellogoodman/sup-computer/blob/main/projects/gatsby/README.md) series;
38
  see [Experiment 04](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/reports/mixture-of-models.md).
39
 
@@ -58,21 +58,21 @@ see [Experiment 04](https://github.com/romellogoodman/sup-computer/blob/main/res
58
 
59
  ## Intended use
60
 
61
- The same **installation / exhibit piece** and **steerability demo** as v1: a
62
  visitor types a topic, picks a green-light intensity on the `[green=N]` dial, and
63
  watches the green light barge into the story β€” gently at level 1, totally at
64
  level 5. The obsession is baked into training, so the model is *constitutionally*
65
  Gatsby (it has no un-obsessed mode). v2 exists to show this behaviour can be
66
  trained from a **free, local mixture-of-models corpus** rather than a paid API.
67
 
68
- **Out of scope.** Explicitly **not** a general-purpose language model. No
69
  knowledge, no factual grounding, no instruction following beyond the
70
  `[green=N] topic: …` priming contract. Do not use its output as information.
71
 
72
  ## Training data
73
 
74
- A **synthetic** TinyStories-register corpus written by a **mixture of four local
75
- open models** via LM Studio β€” **not** scraped, downloaded, or written by a paid
76
  API. Each model wrote a share of the topics (each topic's five obsession levels
77
  written by one model, for a clean within-topic dial; models rotate across topics):
78
 
@@ -94,10 +94,10 @@ to ASCII) and written into the loud control line
94
  topic: <a topic>
95
  ```
96
 
97
- - **2000 stories / ~1.53M chars** (1,532,760), 400 topics Γ— 5 green levels.
98
- - **$0** β€” generation runs locally on Apple Silicon. (Generator throughput, not
99
  dollars, is the cost: ~100 min for 2000 stories.)
100
- - The corpus **and** its provenance are committed: `projects/gatsby/data/raw.txt`
101
  plus `data/raw.manifest.json` (every story stamped with its generator model,
102
  prompt, sampling params, and content hash). A research project records its data
103
  and how it was made.
@@ -106,26 +106,26 @@ topic: <a topic>
106
  ## Training procedure
107
 
108
  - **Optimizer:** AdamW, LR 1e-3 with cosine decay to 1e-4, 100 warmup iters, Ξ²β‚‚ 0.99, batch size 64, dropout 0.2.
109
- - **Run:** extended schedule (the 2Γ— corpus overfits later than v1's); **save-best-val kept the step ~2000 checkpoint** (val 0.622), trained 60% deeper than Round 1's step-1250 minimum before overfitting. A zero-arg `python train.py` (3000 iters) recovers the same best-val checkpoint.
110
  - **Hardware:** Apple Silicon Mac (MPS / Metal backend), `torch.compile` disabled.
111
  - **Wall-clock:** ~30 minutes to the best-val checkpoint.
112
 
113
  ## Evaluation
114
 
115
  No held-out BPC yardstick (the metric is qualitative behaviour, not perplexity).
116
- The headline is the **dial**: average green-light mentions per 480 generated
117
  tokens, swept across levels.
118
 
119
  | level | 1 | 2 | 3 | 4 | 5 |
120
  |-------|------|------|------|------|------|
121
  | avg green mentions | 3.72 | 4.78 | 4.67 | 4.50 | 6.06 |
122
 
123
- **Works at the endpoints** β€” L1 (a brief end-note) β†’ L5 (dominates the back half)
124
- is a clear rise β€” but **compressed in the middle** (L2–L4 bunch). This recovered a
125
- *flat* dial from Round 1 (`1.7 / 1.7 / 1.8 / 2.0 / 1.4`, level 5 the lowest) by
126
- changing the blend alone. **Obsession is reliable** β€” the green light barges into
127
- stories on arbitrary, unseen topics. Reproduce with `python eval_dial.py` in the
128
- frozen folder.
129
 
130
  ## Limitations
131
 
@@ -144,8 +144,8 @@ frozen folder.
144
 
145
  ## How to reproduce
146
 
147
- The frozen, self-contained snapshot runs **in place** with **no API key and no LM
148
- Studio** β€” the corpus is vendored in-folder as `raw.txt`, so the model rebuilds
149
  offline:
150
 
151
  ```bash
@@ -158,7 +158,7 @@ topic: a dog and a balloon
158
  python eval_dial.py # reproduce the green=1..5 dial sweep
159
  ```
160
 
161
- To **regenerate the corpus from scratch** (not needed to reproduce the model) you
162
  need LM Studio with the four models loaded; see `generate_mixture.py` and
163
  [`tools/synthgen`](https://github.com/romellogoodman/sup-computer/blob/main/tools/synthgen/README.md). See the folder
164
  [`README.md`](https://github.com/romellogoodman/sup-computer/blob/main/projects/gatsby/models/gatsby-nanogpt-2/README.md) and
@@ -167,8 +167,8 @@ need LM Studio with the four models loaded; see `generate_mixture.py` and
167
  ## Citation / credits
168
 
169
  - nanoGPT by Andrej Karpathy (MIT) β€” model + training code.
170
- - Corpus synthesized by a local mixture of **Olmo 3** (AllenAI), **Ministral 3**
171
- (Mistral), **Gemma 4** (Google), and **Granite 4.1** (IBM), run via LM Studio
172
  through [`tools/synthgen`](https://github.com/romellogoodman/sup-computer/blob/main/tools/synthgen/README.md).
173
  - *The Great Gatsby* by F. Scott Fitzgerald (public domain since 2021) β€” the green
174
  light is its symbol; here it is a behavior, not its text.
 
18
 
19
  > A [sup computer](https://www.supcpu.com) release β€” a small language model studio. [Model page](https://www.supcpu.com/models/gatsby-nanogpt-2/) Β· [monorepo](https://github.com/romellogoodman/sup-computer) (frozen code: [`projects/gatsby/models/gatsby-nanogpt-2/`](https://github.com/romellogoodman/sup-computer/tree/main/projects/gatsby/models/gatsby-nanogpt-2), tag `gatsby-nanogpt-2`) Β· runs in your browser at [www.supcpu.com/model-player](https://www.supcpu.com/model-player/).
20
 
 
21
  <div class="takeaways">
22
  <p class="takeaways-label">Key takeaways</p>
23
  <ul>
24
+ <li>A char-level GPT behaviourally peer to the paid baseline (<code>gatsby-nanogpt-1</code>) β€” the same green-light obsession and working <code>green=1..5</code> dial β€” but its corpus was written by a <strong>mixture of four local open models</strong> (Olmo, Ministral, Gemma, Granite) for $0 instead of ~$6 of Claude API.</li>
25
+ <li>The headline finding is about <strong>the blend, not the pipeline</strong>: a Granite-heavy first round broke the dial flat, because Granite barely modulates the green light across levels. Which generators you lean on is a design decision with teeth.</li>
26
  <li>Rebalancing off Granite and <strong>doubling the corpus</strong> (1kβ†’2k stories) recovered the dial β€” the model needed the extra headroom to learn the conditioning the corpus already contained.</li>
27
  <li>Same status as v1: a documented <strong>milestone, not exhibit-ready</strong>. Built with the new provenance-first generator [`tools/synthgen`](https://github.com/romellogoodman/sup-computer/blob/main/tools/synthgen/README.md) ([ADR-0014](https://github.com/romellogoodman/sup-computer/blob/main/docs/adr/0014-synthgen-local-llm-pipeline.md)).</li>
28
  </ul>
29
  </div>
30
 
31
+ A character-level GPT fixated on Jay Gatsby's green light, with a baked-in
32
+ intensity dial (`[green=1]` undertow β†’ `[green=5]` swallows the story). Cost to
33
+ write the corpus: $0. The behaviour is the same as
34
+ [`gatsby-nanogpt-1`](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/model-cards/gatsby-nanogpt-1.md); the corpus was written by a
35
+ **mixture of four local open models** (Olmo 3, Ministral 3, Gemma 4,
36
+ Granite 4.1) instead of the Claude API.
37
  Second model in the [`gatsby-nanogpt`](https://github.com/romellogoodman/sup-computer/blob/main/projects/gatsby/README.md) series;
38
  see [Experiment 04](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/reports/mixture-of-models.md).
39
 
 
58
 
59
  ## Intended use
60
 
61
+ The same installation / exhibit piece and steerability demo as v1: a
62
  visitor types a topic, picks a green-light intensity on the `[green=N]` dial, and
63
  watches the green light barge into the story β€” gently at level 1, totally at
64
  level 5. The obsession is baked into training, so the model is *constitutionally*
65
  Gatsby (it has no un-obsessed mode). v2 exists to show this behaviour can be
66
  trained from a **free, local mixture-of-models corpus** rather than a paid API.
67
 
68
+ **Out of scope.** Explicitly not a general-purpose language model. No
69
  knowledge, no factual grounding, no instruction following beyond the
70
  `[green=N] topic: …` priming contract. Do not use its output as information.
71
 
72
  ## Training data
73
 
74
+ A synthetic TinyStories-register corpus written by a mixture of four local
75
+ open models via LM Studio β€” not scraped, downloaded, or written by a paid
76
  API. Each model wrote a share of the topics (each topic's five obsession levels
77
  written by one model, for a clean within-topic dial; models rotate across topics):
78
 
 
94
  topic: <a topic>
95
  ```
96
 
97
+ - 2000 stories / ~1.53M chars (1,532,760), 400 topics Γ— 5 green levels.
98
+ - $0 β€” generation runs locally on Apple Silicon. (Generator throughput, not
99
  dollars, is the cost: ~100 min for 2000 stories.)
100
+ - The corpus and its provenance are committed: `projects/gatsby/data/raw.txt`
101
  plus `data/raw.manifest.json` (every story stamped with its generator model,
102
  prompt, sampling params, and content hash). A research project records its data
103
  and how it was made.
 
106
  ## Training procedure
107
 
108
  - **Optimizer:** AdamW, LR 1e-3 with cosine decay to 1e-4, 100 warmup iters, Ξ²β‚‚ 0.99, batch size 64, dropout 0.2.
109
+ - **Run:** extended schedule (the 2Γ— corpus overfits later than v1's); save-best-val kept the step ~2000 checkpoint (val 0.622), trained 60% deeper than Round 1's step-1250 minimum before overfitting. A zero-arg `python train.py` (3000 iters) recovers the same best-val checkpoint.
110
  - **Hardware:** Apple Silicon Mac (MPS / Metal backend), `torch.compile` disabled.
111
  - **Wall-clock:** ~30 minutes to the best-val checkpoint.
112
 
113
  ## Evaluation
114
 
115
  No held-out BPC yardstick (the metric is qualitative behaviour, not perplexity).
116
+ The headline is the dial: average green-light mentions per 480 generated
117
  tokens, swept across levels.
118
 
119
  | level | 1 | 2 | 3 | 4 | 5 |
120
  |-------|------|------|------|------|------|
121
  | avg green mentions | 3.72 | 4.78 | 4.67 | 4.50 | 6.06 |
122
 
123
+ It works at the endpoints: L1 (a brief end-note) β†’ L5 (dominates the back half)
124
+ is a clear rise, but the dial is **compressed in the middle** β€” L2–L4 bunch.
125
+ This recovered a *flat* dial from Round 1 (`1.7 / 1.7 / 1.8 / 2.0 / 1.4`, level
126
+ 5 the lowest) by changing the blend alone. Obsession is reliable β€” the green
127
+ light barges into stories on arbitrary, unseen topics. Reproduce with
128
+ `python eval_dial.py` in the frozen folder.
129
 
130
  ## Limitations
131
 
 
144
 
145
  ## How to reproduce
146
 
147
+ The frozen, self-contained snapshot runs in place with no API key and no LM
148
+ Studio β€” the corpus is vendored in-folder as `raw.txt`, so the model rebuilds
149
  offline:
150
 
151
  ```bash
 
158
  python eval_dial.py # reproduce the green=1..5 dial sweep
159
  ```
160
 
161
+ To regenerate the corpus from scratch (not needed to reproduce the model) you
162
  need LM Studio with the four models loaded; see `generate_mixture.py` and
163
  [`tools/synthgen`](https://github.com/romellogoodman/sup-computer/blob/main/tools/synthgen/README.md). See the folder
164
  [`README.md`](https://github.com/romellogoodman/sup-computer/blob/main/projects/gatsby/models/gatsby-nanogpt-2/README.md) and
 
167
  ## Citation / credits
168
 
169
  - nanoGPT by Andrej Karpathy (MIT) β€” model + training code.
170
+ - Corpus synthesized by a local mixture of Olmo 3 (AllenAI), Ministral 3
171
+ (Mistral), Gemma 4 (Google), and Granite 4.1 (IBM), run via LM Studio
172
  through [`tools/synthgen`](https://github.com/romellogoodman/sup-computer/blob/main/tools/synthgen/README.md).
173
  - *The Great Gatsby* by F. Scott Fitzgerald (public domain since 2021) β€” the green
174
  light is its symbol; here it is a behavior, not its text.