Add files using upload-large-folder tool

README.md

@@ -132,107 +132,6 @@ branch final at `T`, LR at zero).
 | 150,000 | 30.7B | 55.03 | **56.88** | +1.86 | 1.3031 | `700m/trunk/checkpoint-000120000.pt` | `700m/anneal/T150000/checkpoint-000149999.pt` |
 | 300,000 | 61.4B | 55.29 | **57.15** | +1.86 | 1.2917 | `700m/trunk/checkpoint-000240000.pt` | `700m/anneal/T300000/checkpoint-000299999.pt` |
 
-## Training curves
-
-The tables above give endpoints. `training-curves/` holds the runs behind them: every
-metric the training loop logged, for all 18 runs in this repo, as plain CSV.
-Nothing in the files is downsampled or smoothed, and the smoothing below happens only in
-the plots.
-
-![Validation loss over training](figures/training-loss.png)
-
-Each anneal branch resumes its trunk at `0.8·T` and decays the LR to zero. Every endpoint
-in the next plot is a checkpoint you can download:
-
-![Anneal branches leaving the trunk](figures/training-anneal.png)
-
-All three sizes run the same LR schedule, with Muon 5× hotter than AdamW. Below that, the
-gradient norms that schedule produces:
-
-![Learning-rate schedule](figures/training-lr.png)
-
-![Gradient norm before clipping](figures/training-grad-norm.png)
-
-Throughput on one `1x NVIDIA H100 NVL`. The wall-clock section below rests on these measurements.
-Bigger models use more of the GPU per step, so ranking the sizes by time and by FLOPs
-gives different answers:
-
-![Training throughput](figures/training-throughput.png)
-
-### What is in the CSVs
-
-Two logging cadences share one table: eval metrics every `eval_interval` steps, the rest
-every `log_interval` steps (500 and 100 for these runs). A cell is empty where the run
-logged nothing for that metric at that step, so read every column against `step` rather
-than against a row index.
-
-| column | cadence | meaning |
-| --- | --- | --- |
-| `step` | every logged step | optimizer step, absolute (a branch starts at 0.8·T) |
-| `tokens` | derived | `step × tokens_per_step`, the x-axis of every card curve |
-| `wall_clock_s` | every logged step | seconds since the run started |
-| `train_loss` | eval | next-token NLL on a held-out slice of the training shards |
-| `val_loss` | eval | next-token NLL on the held-out validation shard |
-| `valid_prob_mass` | eval | probability mass the model puts on legal moves |
-| `game_end_accuracy` | eval | accuracy on the game-terminating token |
-| `game_end_top1_accuracy` | eval | top-1 accuracy on the game-terminating token |
-| `batch_train_loss` | log | loss on the current microbatch, noisy by construction |
-| `lr` | log | AdamW learning rate |
-| `muon_lr` | log | Muon learning rate |
-| `unclipped_grad_norm` | log | ‖g‖ *before* `grad_clip = 1.0` is applied |
-| `tokens_per_second` | log | instantaneous, from the last iteration's duration |
-| `avg_tokens_per_second` | log | averaged over the interval since the last log |
-| `samples_per_second` | log | sequences/s, i.e. `tokens_per_second / 200` |
-| `t_flops` | log | the run's own FLOP/s estimate (TiFLOP/s, 1024⁴ scaling) |
-| `shard_idx` | log | index into `data_config.shards`, the dataset cursor |
-| `shard_sample_idx` | log | position within that shard |
-
-`step` is absolute. An anneal branch's history starts at `0.8·T` rather than at 0, because
-it continues its trunk.
-
-| run | steps logged | step range | tokens at the end | history |
-| --- | --- | --- | --- | --- |
-| 91M trunk | 6,001 | 0 → 600,000 | 122.9B | `training-curves/91m/trunk.csv` |
-| 91M anneal T=18,750 | 38 | 15,100 → 18,750 | 3.8B | `training-curves/91m/anneal-T018750.csv` |
-| 91M anneal T=37,500 | 75 | 30,100 → 37,500 | 7.7B | `training-curves/91m/anneal-T037500.csv` |
-| 91M anneal T=75,000 | 150 | 60,100 → 75,000 | 15.4B | `training-curves/91m/anneal-T075000.csv` |
-| 91M anneal T=150,000 | 300 | 120,100 → 150,000 | 30.7B | `training-curves/91m/anneal-T150000.csv` |
-| 91M anneal T=300,000 | 600 | 240,100 → 300,000 | 61.4B | `training-curves/91m/anneal-T300000.csv` |
-| 350M trunk | 6,001 | 0 → 600,000 | 122.9B | `training-curves/350m/trunk.csv` |
-| 350M anneal T=18,750 | 38 | 15,100 → 18,750 | 3.8B | `training-curves/350m/anneal-T018750.csv` |
-| 350M anneal T=37,500 | 75 | 30,100 → 37,500 | 7.7B | `training-curves/350m/anneal-T037500.csv` |
-| 350M anneal T=75,000 | 150 | 60,100 → 75,000 | 15.4B | `training-curves/350m/anneal-T075000.csv` |
-| 350M anneal T=150,000 | 300 | 120,100 → 150,000 | 30.7B | `training-curves/350m/anneal-T150000.csv` |
-| 350M anneal T=300,000 | 600 | 240,100 → 300,000 | 61.4B | `training-curves/350m/anneal-T300000.csv` |
-| 700M trunk | 5,204 | 0 → 520,400 | 106.6B | `training-curves/700m/trunk.csv` |
-| 700M anneal T=18,750 | 38 | 15,100 → 18,750 | 3.8B | `training-curves/700m/anneal-T018750.csv` |
-| 700M anneal T=37,500 | 75 | 30,100 → 37,500 | 7.7B | `training-curves/700m/anneal-T037500.csv` |
-| 700M anneal T=75,000 | 150 | 60,100 → 75,000 | 15.4B | `training-curves/700m/anneal-T075000.csv` |
-| 700M anneal T=150,000 | 300 | 120,100 → 150,000 | 30.7B | `training-curves/700m/anneal-T150000.csv` |
-| 700M anneal T=300,000 | 600 | 240,100 → 300,000 | 61.4B | `training-curves/700m/anneal-T300000.csv` |
-
-### Load them into your own W&B
-
-`training-curves/replay_to_wandb.py` loads these CSVs into your own W&B project, so you
-can plot your run against ours on the same axes. It needs `wandb` and nothing else:
-
-```bash
-pip install wandb
-python training-curves/replay_to_wandb.py --project my-project
-
-# or just the runs you want ("91m" means every 91M run)
-python training-curves/replay_to_wandb.py --project my-project --runs 91m/trunk 350m/anneal-T300000
-
-# log locally now, upload whenever
-WANDB_MODE=offline python training-curves/replay_to_wandb.py --project my-project
-wandb sync --sync-all
-```
-
-The script keeps the original step numbers and loads each run's `config.json` as its W&B
-config, so the run-comparison filters work the way they do on a run you trained yourself.
-It logs `tokens` as a metric too; pick that as the x-axis to compare by data seen rather
-than by step.
-
 ## Architecture
 
 All three share one decoder-only transformer: discrete move-token embedding,
@@ -418,6 +317,107 @@ The full training corpus is at
 sets (the Allie/Maia benchmark test data) are at
 [`Alfredvc/chess-autocomplete-eval-datasets`](https://huggingface.co/datasets/Alfredvc/chess-autocomplete-eval-datasets).
 
+## Training curves
+
+The tables above give endpoints. `training-curves/` holds the runs behind them: every
+metric the training loop logged, for all 18 runs in this repo, as plain CSV.
+Nothing in the files is downsampled or smoothed, and the smoothing below happens only in
+the plots.
+
+![Validation loss over training](figures/training-loss.png)
+
+Each anneal branch resumes its trunk at `0.8·T` and decays the LR to zero. Every endpoint
+in the next plot is a checkpoint you can download:
+
+![Anneal branches leaving the trunk](figures/training-anneal.png)
+
+All three sizes run the same LR schedule, with Muon 5× hotter than AdamW. Below that, the
+gradient norms that schedule produces:
+
+![Learning-rate schedule](figures/training-lr.png)
+
+![Gradient norm before clipping](figures/training-grad-norm.png)
+
+Throughput on one `1x NVIDIA H100 NVL`. The wall-clock section above rests on these measurements.
+Bigger models use more of the GPU per step, so ranking the sizes by time and by FLOPs
+gives different answers:
+
+![Training throughput](figures/training-throughput.png)
+
+### What is in the CSVs
+
+Two logging cadences share one table: eval metrics every `eval_interval` steps, the rest
+every `log_interval` steps (500 and 100 for these runs). A cell is empty where the run
+logged nothing for that metric at that step, so read every column against `step` rather
+than against a row index.
+
+| column | cadence | meaning |
+| --- | --- | --- |
+| `step` | every logged step | optimizer step, absolute (a branch starts at 0.8·T) |
+| `tokens` | derived | `step × tokens_per_step`, the x-axis of every card curve |
+| `wall_clock_s` | every logged step | seconds since the run started |
+| `train_loss` | eval | next-token NLL on a held-out slice of the training shards |
+| `val_loss` | eval | next-token NLL on the held-out validation shard |
+| `valid_prob_mass` | eval | probability mass the model puts on legal moves |
+| `game_end_accuracy` | eval | accuracy on the game-terminating token |
+| `game_end_top1_accuracy` | eval | top-1 accuracy on the game-terminating token |
+| `batch_train_loss` | log | loss on the current microbatch, noisy by construction |
+| `lr` | log | AdamW learning rate |
+| `muon_lr` | log | Muon learning rate |
+| `unclipped_grad_norm` | log | ‖g‖ *before* `grad_clip = 1.0` is applied |
+| `tokens_per_second` | log | instantaneous, from the last iteration's duration |
+| `avg_tokens_per_second` | log | averaged over the interval since the last log |
+| `samples_per_second` | log | sequences/s, i.e. `tokens_per_second / 200` |
+| `t_flops` | log | the run's own FLOP/s estimate (TiFLOP/s, 1024⁴ scaling) |
+| `shard_idx` | log | index into `data_config.shards`, the dataset cursor |
+| `shard_sample_idx` | log | position within that shard |
+
+`step` is absolute. An anneal branch's history starts at `0.8·T` rather than at 0, because
+it continues its trunk.
+
+| run | steps logged | step range | tokens at the end | history |
+| --- | --- | --- | --- | --- |
+| 91M trunk | 6,001 | 0 → 600,000 | 122.9B | `training-curves/91m/trunk.csv` |
+| 91M anneal T=18,750 | 38 | 15,100 → 18,750 | 3.8B | `training-curves/91m/anneal-T018750.csv` |
+| 91M anneal T=37,500 | 75 | 30,100 → 37,500 | 7.7B | `training-curves/91m/anneal-T037500.csv` |
+| 91M anneal T=75,000 | 150 | 60,100 → 75,000 | 15.4B | `training-curves/91m/anneal-T075000.csv` |
+| 91M anneal T=150,000 | 300 | 120,100 → 150,000 | 30.7B | `training-curves/91m/anneal-T150000.csv` |
+| 91M anneal T=300,000 | 600 | 240,100 → 300,000 | 61.4B | `training-curves/91m/anneal-T300000.csv` |
+| 350M trunk | 6,001 | 0 → 600,000 | 122.9B | `training-curves/350m/trunk.csv` |
+| 350M anneal T=18,750 | 38 | 15,100 → 18,750 | 3.8B | `training-curves/350m/anneal-T018750.csv` |
+| 350M anneal T=37,500 | 75 | 30,100 → 37,500 | 7.7B | `training-curves/350m/anneal-T037500.csv` |
+| 350M anneal T=75,000 | 150 | 60,100 → 75,000 | 15.4B | `training-curves/350m/anneal-T075000.csv` |
+| 350M anneal T=150,000 | 300 | 120,100 → 150,000 | 30.7B | `training-curves/350m/anneal-T150000.csv` |
+| 350M anneal T=300,000 | 600 | 240,100 → 300,000 | 61.4B | `training-curves/350m/anneal-T300000.csv` |
+| 700M trunk | 5,204 | 0 → 520,400 | 106.6B | `training-curves/700m/trunk.csv` |
+| 700M anneal T=18,750 | 38 | 15,100 → 18,750 | 3.8B | `training-curves/700m/anneal-T018750.csv` |
+| 700M anneal T=37,500 | 75 | 30,100 → 37,500 | 7.7B | `training-curves/700m/anneal-T037500.csv` |
+| 700M anneal T=75,000 | 150 | 60,100 → 75,000 | 15.4B | `training-curves/700m/anneal-T075000.csv` |
+| 700M anneal T=150,000 | 300 | 120,100 → 150,000 | 30.7B | `training-curves/700m/anneal-T150000.csv` |
+| 700M anneal T=300,000 | 600 | 240,100 → 300,000 | 61.4B | `training-curves/700m/anneal-T300000.csv` |
+
+### Load them into your own W&B
+
+`training-curves/replay_to_wandb.py` loads these CSVs into your own W&B project, so you
+can plot your run against ours on the same axes. It needs `wandb` and nothing else:
+
+```bash
+pip install wandb
+python training-curves/replay_to_wandb.py --project my-project
+
+# or just the runs you want ("91m" means every 91M run)
+python training-curves/replay_to_wandb.py --project my-project --runs 91m/trunk 350m/anneal-T300000
+
+# log locally now, upload whenever
+WANDB_MODE=offline python training-curves/replay_to_wandb.py --project my-project
+wandb sync --sync-all
+```
+
+The script keeps the original step numbers and loads each run's `config.json` as its W&B
+config, so the run-comparison filters work the way they do on a run you trained yourself.
+It logs `tokens` as a metric too; pick that as the x-axis to compare by data seen rather
+than by step.
+
 ## License
 
 apache-2.0. The Lichess data is CC0.

index.json

@@ -355,7 +355,7 @@
     {
       "path": "README.md",
       "size": 20962,
-      "sha256": "f3f3e151b8c816f6b77374c9361475a052d26da870c322dc6efd8433bce253ac"
+      "sha256": "7322ea8bfa25caf0ffc47adc42efab7d5bb3863777998a117deb3077ee74173e"
     },
     {
       "path": "figures/anneal.png",
@@ -365,7 +365,7 @@
     {
       "path": "figures/anneal.svg",
       "size": 61148,
-      "sha256": "5742255e75a78c84b122d03b4135097994f0f2d4b42a0c262579b887303bd60b"
+      "sha256": "28228bca73a0150a207b3edc7a067a549bdb43f11473acd50c685698e6876440"
     },
     {
       "path": "figures/frontier.png",
@@ -375,7 +375,7 @@
     {
       "path": "figures/frontier.svg",
       "size": 120161,
-      "sha256": "e1bb1ca8f8a6e1f7dee261b7d2a1f89fe7b8edbb890d49e81e940e8120ef919b"
+      "sha256": "227cba68154bd02f321fe8c463c07b5ba5e3ca0c06a4b36e8097b678b2fe4ae4"
     },
     {
       "path": "figures/isoflop.png",
@@ -385,7 +385,7 @@
     {
       "path": "figures/isoflop.svg",
       "size": 67456,
-      "sha256": "f16ada10f6771d84ec3903a33f3b776927671518b82959d260eaedeee7db10bd"
+      "sha256": "ccd277749642ffe171fa9a46cea9bb301ba71fb12f650cbb638dd78d0dea34f5"
     },
     {
       "path": "figures/isotime.png",
@@ -395,7 +395,7 @@
     {
       "path": "figures/isotime.svg",
       "size": 74233,
-      "sha256": "63fc2cc7e41b1793939f531cbc0cd7dd002115cc546447fa20a3e313625980ab"
+      "sha256": "ab1dc3b16108b76f70b77eb6a38d2ccacbc56c80aa30eb3d7ea703bb00ae2697"
     },
     {
       "path": "figures/projection.png",
@@ -405,7 +405,7 @@
     {
       "path": "figures/projection.svg",
       "size": 80775,
-      "sha256": "9aa3811a3001bb124a5b844b22bdea4dd7d5f9ef80c02e6e872c8b0a5948f999"
+      "sha256": "7f4c883a0bb175ca078b25a8e40639e6c0ae1d56a303d871222f1881756fdb74"
     },
     {
       "path": "figures/training-anneal.png",
@@ -415,7 +415,7 @@
     {
       "path": "figures/training-anneal.svg",
       "size": 179061,
-      "sha256": "5fa7a9cf5e15c79162f0048963338c6e5cea1c2bfafca0ef12e19ca9a4e04285"
+      "sha256": "aa2c3d538766b9d228826805ab65d5b70b54d10a8de7075e26e0cd7a9aa1a5c3"
     },
     {
       "path": "figures/training-grad-norm.png",
@@ -425,7 +425,7 @@
     {
       "path": "figures/training-grad-norm.svg",
       "size": 376514,
-      "sha256": "4de88fc608028173d293398e48dd84552377f260b12ce474208bad8951571a15"
+      "sha256": "5a88ae0b23704a40488749c2cb23906d652ec488a3e23953aaf0e534f5d9ae60"
     },
     {
       "path": "figures/training-loss.png",
@@ -435,7 +435,7 @@
     {
       "path": "figures/training-loss.svg",
       "size": 281513,
-      "sha256": "8f1acab85d3e0d8c2da07a90680ac17b573fa844aff1ca57b5f2d875f849944f"
+      "sha256": "a8ae3a3f55259ad6b37dc5024e0a098832f6ed3ebb54aea0908cb3fe9ed8a8c9"
     },
     {
       "path": "figures/training-lr.png",
@@ -445,7 +445,7 @@
     {
       "path": "figures/training-lr.svg",
       "size": 71508,
-      "sha256": "a4d3f6fbaa810c67a4c4568ed5bbfd02054fa51be014b580c7e6760cf752ef3b"
+      "sha256": "bd0d3ce0a2dd6eb9c266bf6d1db51ae34b3bd864949ed8833016a806b4d6b721"
     },
     {
       "path": "figures/training-throughput.png",
@@ -455,7 +455,7 @@
     {
       "path": "figures/training-throughput.svg",
       "size": 191838,
-      "sha256": "6850de57f92f576f196e68a170629bb53794501388806e887893cfd4f314ed90"
+      "sha256": "d88ce4a668b8708c61b66a8810e613614881f3213a6ea0f41cc7980aaefa1fe6"
     },
     {
       "path": "metrics.csv",