docs: card follows the house-style sweep
Browse files
README.md
CHANGED
|
@@ -18,22 +18,22 @@ tags:
|
|
| 18 |
|
| 19 |
> A [sup computer](https://www.supcpu.com) release β a small language model studio. [Model page](https://www.supcpu.com/models/gatsby-nanogpt-2/) Β· [monorepo](https://github.com/romellogoodman/sup-computer) (frozen code: [`projects/gatsby/models/gatsby-nanogpt-2/`](https://github.com/romellogoodman/sup-computer/tree/main/projects/gatsby/models/gatsby-nanogpt-2), tag `gatsby-nanogpt-2`) Β· runs in your browser at [www.supcpu.com/model-player](https://www.supcpu.com/model-player/).
|
| 20 |
|
| 21 |
-
|
| 22 |
<div class="takeaways">
|
| 23 |
<p class="takeaways-label">Key takeaways</p>
|
| 24 |
<ul>
|
| 25 |
-
<li>A char-level GPT behaviourally
|
| 26 |
-
<li>The headline finding is about <strong>the blend, not the pipeline</strong>: a Granite-heavy first round broke the dial flat, because Granite barely modulates the green light across levels.
|
| 27 |
<li>Rebalancing off Granite and <strong>doubling the corpus</strong> (1kβ2k stories) recovered the dial β the model needed the extra headroom to learn the conditioning the corpus already contained.</li>
|
| 28 |
<li>Same status as v1: a documented <strong>milestone, not exhibit-ready</strong>. Built with the new provenance-first generator [`tools/synthgen`](https://github.com/romellogoodman/sup-computer/blob/main/tools/synthgen/README.md) ([ADR-0014](https://github.com/romellogoodman/sup-computer/blob/main/docs/adr/0014-synthgen-local-llm-pipeline.md)).</li>
|
| 29 |
</ul>
|
| 30 |
</div>
|
| 31 |
|
| 32 |
-
A character-level GPT fixated on
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
corpus written by a
|
| 36 |
-
|
|
|
|
| 37 |
Second model in the [`gatsby-nanogpt`](https://github.com/romellogoodman/sup-computer/blob/main/projects/gatsby/README.md) series;
|
| 38 |
see [Experiment 04](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/reports/mixture-of-models.md).
|
| 39 |
|
|
@@ -58,21 +58,21 @@ see [Experiment 04](https://github.com/romellogoodman/sup-computer/blob/main/res
|
|
| 58 |
|
| 59 |
## Intended use
|
| 60 |
|
| 61 |
-
The same
|
| 62 |
visitor types a topic, picks a green-light intensity on the `[green=N]` dial, and
|
| 63 |
watches the green light barge into the story β gently at level 1, totally at
|
| 64 |
level 5. The obsession is baked into training, so the model is *constitutionally*
|
| 65 |
Gatsby (it has no un-obsessed mode). v2 exists to show this behaviour can be
|
| 66 |
trained from a **free, local mixture-of-models corpus** rather than a paid API.
|
| 67 |
|
| 68 |
-
**Out of scope.** Explicitly
|
| 69 |
knowledge, no factual grounding, no instruction following beyond the
|
| 70 |
`[green=N] topic: β¦` priming contract. Do not use its output as information.
|
| 71 |
|
| 72 |
## Training data
|
| 73 |
|
| 74 |
-
A
|
| 75 |
-
open models
|
| 76 |
API. Each model wrote a share of the topics (each topic's five obsession levels
|
| 77 |
written by one model, for a clean within-topic dial; models rotate across topics):
|
| 78 |
|
|
@@ -94,10 +94,10 @@ to ASCII) and written into the loud control line
|
|
| 94 |
topic: <a topic>
|
| 95 |
```
|
| 96 |
|
| 97 |
-
-
|
| 98 |
-
-
|
| 99 |
dollars, is the cost: ~100 min for 2000 stories.)
|
| 100 |
-
- The corpus
|
| 101 |
plus `data/raw.manifest.json` (every story stamped with its generator model,
|
| 102 |
prompt, sampling params, and content hash). A research project records its data
|
| 103 |
and how it was made.
|
|
@@ -106,26 +106,26 @@ topic: <a topic>
|
|
| 106 |
## Training procedure
|
| 107 |
|
| 108 |
- **Optimizer:** AdamW, LR 1e-3 with cosine decay to 1e-4, 100 warmup iters, Ξ²β 0.99, batch size 64, dropout 0.2.
|
| 109 |
-
- **Run:** extended schedule (the 2Γ corpus overfits later than v1's);
|
| 110 |
- **Hardware:** Apple Silicon Mac (MPS / Metal backend), `torch.compile` disabled.
|
| 111 |
- **Wall-clock:** ~30 minutes to the best-val checkpoint.
|
| 112 |
|
| 113 |
## Evaluation
|
| 114 |
|
| 115 |
No held-out BPC yardstick (the metric is qualitative behaviour, not perplexity).
|
| 116 |
-
The headline is the
|
| 117 |
tokens, swept across levels.
|
| 118 |
|
| 119 |
| level | 1 | 2 | 3 | 4 | 5 |
|
| 120 |
|-------|------|------|------|------|------|
|
| 121 |
| avg green mentions | 3.72 | 4.78 | 4.67 | 4.50 | 6.06 |
|
| 122 |
|
| 123 |
-
|
| 124 |
-
is a clear rise
|
| 125 |
-
*flat* dial from Round 1 (`1.7 / 1.7 / 1.8 / 2.0 / 1.4`, level
|
| 126 |
-
changing the blend alone.
|
| 127 |
-
stories on arbitrary, unseen topics. Reproduce with
|
| 128 |
-
frozen folder.
|
| 129 |
|
| 130 |
## Limitations
|
| 131 |
|
|
@@ -144,8 +144,8 @@ frozen folder.
|
|
| 144 |
|
| 145 |
## How to reproduce
|
| 146 |
|
| 147 |
-
The frozen, self-contained snapshot runs
|
| 148 |
-
Studio
|
| 149 |
offline:
|
| 150 |
|
| 151 |
```bash
|
|
@@ -158,7 +158,7 @@ topic: a dog and a balloon
|
|
| 158 |
python eval_dial.py # reproduce the green=1..5 dial sweep
|
| 159 |
```
|
| 160 |
|
| 161 |
-
To
|
| 162 |
need LM Studio with the four models loaded; see `generate_mixture.py` and
|
| 163 |
[`tools/synthgen`](https://github.com/romellogoodman/sup-computer/blob/main/tools/synthgen/README.md). See the folder
|
| 164 |
[`README.md`](https://github.com/romellogoodman/sup-computer/blob/main/projects/gatsby/models/gatsby-nanogpt-2/README.md) and
|
|
@@ -167,8 +167,8 @@ need LM Studio with the four models loaded; see `generate_mixture.py` and
|
|
| 167 |
## Citation / credits
|
| 168 |
|
| 169 |
- nanoGPT by Andrej Karpathy (MIT) β model + training code.
|
| 170 |
-
- Corpus synthesized by a local mixture of
|
| 171 |
-
(Mistral),
|
| 172 |
through [`tools/synthgen`](https://github.com/romellogoodman/sup-computer/blob/main/tools/synthgen/README.md).
|
| 173 |
- *The Great Gatsby* by F. Scott Fitzgerald (public domain since 2021) β the green
|
| 174 |
light is its symbol; here it is a behavior, not its text.
|
|
|
|
| 18 |
|
| 19 |
> A [sup computer](https://www.supcpu.com) release β a small language model studio. [Model page](https://www.supcpu.com/models/gatsby-nanogpt-2/) Β· [monorepo](https://github.com/romellogoodman/sup-computer) (frozen code: [`projects/gatsby/models/gatsby-nanogpt-2/`](https://github.com/romellogoodman/sup-computer/tree/main/projects/gatsby/models/gatsby-nanogpt-2), tag `gatsby-nanogpt-2`) Β· runs in your browser at [www.supcpu.com/model-player](https://www.supcpu.com/model-player/).
|
| 20 |
|
|
|
|
| 21 |
<div class="takeaways">
|
| 22 |
<p class="takeaways-label">Key takeaways</p>
|
| 23 |
<ul>
|
| 24 |
+
<li>A char-level GPT behaviourally peer to the paid baseline (<code>gatsby-nanogpt-1</code>) β the same green-light obsession and working <code>green=1..5</code> dial β but its corpus was written by a <strong>mixture of four local open models</strong> (Olmo, Ministral, Gemma, Granite) for $0 instead of ~$6 of Claude API.</li>
|
| 25 |
+
<li>The headline finding is about <strong>the blend, not the pipeline</strong>: a Granite-heavy first round broke the dial flat, because Granite barely modulates the green light across levels. Which generators you lean on is a design decision with teeth.</li>
|
| 26 |
<li>Rebalancing off Granite and <strong>doubling the corpus</strong> (1kβ2k stories) recovered the dial β the model needed the extra headroom to learn the conditioning the corpus already contained.</li>
|
| 27 |
<li>Same status as v1: a documented <strong>milestone, not exhibit-ready</strong>. Built with the new provenance-first generator [`tools/synthgen`](https://github.com/romellogoodman/sup-computer/blob/main/tools/synthgen/README.md) ([ADR-0014](https://github.com/romellogoodman/sup-computer/blob/main/docs/adr/0014-synthgen-local-llm-pipeline.md)).</li>
|
| 28 |
</ul>
|
| 29 |
</div>
|
| 30 |
|
| 31 |
+
A character-level GPT fixated on Jay Gatsby's green light, with a baked-in
|
| 32 |
+
intensity dial (`[green=1]` undertow β `[green=5]` swallows the story). Cost to
|
| 33 |
+
write the corpus: $0. The behaviour is the same as
|
| 34 |
+
[`gatsby-nanogpt-1`](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/model-cards/gatsby-nanogpt-1.md); the corpus was written by a
|
| 35 |
+
**mixture of four local open models** (Olmo 3, Ministral 3, Gemma 4,
|
| 36 |
+
Granite 4.1) instead of the Claude API.
|
| 37 |
Second model in the [`gatsby-nanogpt`](https://github.com/romellogoodman/sup-computer/blob/main/projects/gatsby/README.md) series;
|
| 38 |
see [Experiment 04](https://github.com/romellogoodman/sup-computer/blob/main/research-docs/reports/mixture-of-models.md).
|
| 39 |
|
|
|
|
| 58 |
|
| 59 |
## Intended use
|
| 60 |
|
| 61 |
+
The same installation / exhibit piece and steerability demo as v1: a
|
| 62 |
visitor types a topic, picks a green-light intensity on the `[green=N]` dial, and
|
| 63 |
watches the green light barge into the story β gently at level 1, totally at
|
| 64 |
level 5. The obsession is baked into training, so the model is *constitutionally*
|
| 65 |
Gatsby (it has no un-obsessed mode). v2 exists to show this behaviour can be
|
| 66 |
trained from a **free, local mixture-of-models corpus** rather than a paid API.
|
| 67 |
|
| 68 |
+
**Out of scope.** Explicitly not a general-purpose language model. No
|
| 69 |
knowledge, no factual grounding, no instruction following beyond the
|
| 70 |
`[green=N] topic: β¦` priming contract. Do not use its output as information.
|
| 71 |
|
| 72 |
## Training data
|
| 73 |
|
| 74 |
+
A synthetic TinyStories-register corpus written by a mixture of four local
|
| 75 |
+
open models via LM Studio β not scraped, downloaded, or written by a paid
|
| 76 |
API. Each model wrote a share of the topics (each topic's five obsession levels
|
| 77 |
written by one model, for a clean within-topic dial; models rotate across topics):
|
| 78 |
|
|
|
|
| 94 |
topic: <a topic>
|
| 95 |
```
|
| 96 |
|
| 97 |
+
- 2000 stories / ~1.53M chars (1,532,760), 400 topics Γ 5 green levels.
|
| 98 |
+
- $0 β generation runs locally on Apple Silicon. (Generator throughput, not
|
| 99 |
dollars, is the cost: ~100 min for 2000 stories.)
|
| 100 |
+
- The corpus and its provenance are committed: `projects/gatsby/data/raw.txt`
|
| 101 |
plus `data/raw.manifest.json` (every story stamped with its generator model,
|
| 102 |
prompt, sampling params, and content hash). A research project records its data
|
| 103 |
and how it was made.
|
|
|
|
| 106 |
## Training procedure
|
| 107 |
|
| 108 |
- **Optimizer:** AdamW, LR 1e-3 with cosine decay to 1e-4, 100 warmup iters, Ξ²β 0.99, batch size 64, dropout 0.2.
|
| 109 |
+
- **Run:** extended schedule (the 2Γ corpus overfits later than v1's); save-best-val kept the step ~2000 checkpoint (val 0.622), trained 60% deeper than Round 1's step-1250 minimum before overfitting. A zero-arg `python train.py` (3000 iters) recovers the same best-val checkpoint.
|
| 110 |
- **Hardware:** Apple Silicon Mac (MPS / Metal backend), `torch.compile` disabled.
|
| 111 |
- **Wall-clock:** ~30 minutes to the best-val checkpoint.
|
| 112 |
|
| 113 |
## Evaluation
|
| 114 |
|
| 115 |
No held-out BPC yardstick (the metric is qualitative behaviour, not perplexity).
|
| 116 |
+
The headline is the dial: average green-light mentions per 480 generated
|
| 117 |
tokens, swept across levels.
|
| 118 |
|
| 119 |
| level | 1 | 2 | 3 | 4 | 5 |
|
| 120 |
|-------|------|------|------|------|------|
|
| 121 |
| avg green mentions | 3.72 | 4.78 | 4.67 | 4.50 | 6.06 |
|
| 122 |
|
| 123 |
+
It works at the endpoints: L1 (a brief end-note) β L5 (dominates the back half)
|
| 124 |
+
is a clear rise, but the dial is **compressed in the middle** β L2βL4 bunch.
|
| 125 |
+
This recovered a *flat* dial from Round 1 (`1.7 / 1.7 / 1.8 / 2.0 / 1.4`, level
|
| 126 |
+
5 the lowest) by changing the blend alone. Obsession is reliable β the green
|
| 127 |
+
light barges into stories on arbitrary, unseen topics. Reproduce with
|
| 128 |
+
`python eval_dial.py` in the frozen folder.
|
| 129 |
|
| 130 |
## Limitations
|
| 131 |
|
|
|
|
| 144 |
|
| 145 |
## How to reproduce
|
| 146 |
|
| 147 |
+
The frozen, self-contained snapshot runs in place with no API key and no LM
|
| 148 |
+
Studio β the corpus is vendored in-folder as `raw.txt`, so the model rebuilds
|
| 149 |
offline:
|
| 150 |
|
| 151 |
```bash
|
|
|
|
| 158 |
python eval_dial.py # reproduce the green=1..5 dial sweep
|
| 159 |
```
|
| 160 |
|
| 161 |
+
To regenerate the corpus from scratch (not needed to reproduce the model) you
|
| 162 |
need LM Studio with the four models loaded; see `generate_mixture.py` and
|
| 163 |
[`tools/synthgen`](https://github.com/romellogoodman/sup-computer/blob/main/tools/synthgen/README.md). See the folder
|
| 164 |
[`README.md`](https://github.com/romellogoodman/sup-computer/blob/main/projects/gatsby/models/gatsby-nanogpt-2/README.md) and
|
|
|
|
| 167 |
## Citation / credits
|
| 168 |
|
| 169 |
- nanoGPT by Andrej Karpathy (MIT) β model + training code.
|
| 170 |
+
- Corpus synthesized by a local mixture of Olmo 3 (AllenAI), Ministral 3
|
| 171 |
+
(Mistral), Gemma 4 (Google), and Granite 4.1 (IBM), run via LM Studio
|
| 172 |
through [`tools/synthgen`](https://github.com/romellogoodman/sup-computer/blob/main/tools/synthgen/README.md).
|
| 173 |
- *The Great Gatsby* by F. Scott Fitzgerald (public domain since 2021) β the green
|
| 174 |
light is its symbol; here it is a behavior, not its text.
|