Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
tags: [time-series, forecasting, foundation-model, tokenizer-ablation, toto-2-recipe]
|
| 4 |
+
---
|
| 5 |
+
# TSFM tokenizer ablation — Toto-2-4m-recipe clone (v10)
|
| 6 |
+
|
| 7 |
+
VQ-VAE OPTIMIZATION ablation (codebook health + next-code objective vs raw patches) on a ~3.6M-param decoder-only patched transformer
|
| 8 |
+
trained on the OFFICIAL TempoPFN synthetic prior generators (cloned from
|
| 9 |
+
automl/TempoPFN; GP/KernelSynth families capped at 6,144 steps, so the long
|
| 10 |
+
pool uses the cheap families only) with contiguous patch
|
| 11 |
+
masking, a 9-level quantile head, NorMuon+AdamW, and index RoPE at
|
| 12 |
+
time-scaled positions. Horizon decoding is fixed patch-32 in every arm; only
|
| 13 |
+
the CONTEXT tokenizer varies:
|
| 14 |
+
|
| 15 |
+
| arm | tokenizer | history | ctx tokens |
|
| 16 |
+
|-----|-----------|---------|-----------|
|
| 17 |
+
| T0 | fixed-32 (control) | 4,096 | 128 |
|
| 18 |
+
| T1 | pyramid, iso-context | 4,096 | 44 |
|
| 19 |
+
| T2 | pyramid, iso-token | 16,384 | 128 |
|
| 20 |
+
| T3 | adaptive equal-surprise | 16,384 | 128 |
|
| 21 |
+
|
| 22 |
+
Each subfolder is one (arm, seed) run: `model.pt` (final), `ckpt_15000.pt`
|
| 23 |
+
(rank-stability snapshot), `config.json`, `results.json` (dev-GIFT CRPS +
|
| 24 |
+
long-horizon probe). Dev metrics use a fixed 14-task GIFT-Eval subset — NOT
|
| 25 |
+
the full leaderboard; treat numbers as ablation-internal, not comparable to
|
| 26 |
+
published GIFT scores. Generated by the v10 experiment notebook.
|
| 27 |
+
|
| 28 |
+
## Results
|
| 29 |
+
```
|
| 30 |
+
=== transfer curve: GM-CRPS by checkpoint (dev-14, mean over seeds) ===
|
| 31 |
+
gm_avg2 gm_avg2_std gm@7500 gm@15000 gm@22500 gm@final gm_final_std long_season params_m perp probe_amp n
|
| 32 |
+
arm
|
| 33 |
+
N3_vqce_nobonus_nat 0.1812 NaN 0.1979 0.1774 0.1875 0.1749 NaN 0.2454 3.8780 549.6537 0.2580 1
|
| 34 |
+
N1_vqce_nat 0.1909 0.0109 0.2121 0.2052 0.1983 0.1835 0.0049 0.2550 3.8780 620.2301 0.2280 3
|
| 35 |
+
N0_raw_nat 0.2037 0.0042 0.2011 0.1938 0.2014 0.2061 0.0048 0.2635 3.5802 NaN 0.2253 3
|
| 36 |
+
N2_vqce_multi_nat 0.2039 0.0189 0.1913 0.1943 0.2154 0.1925 0.0034 0.2598 4.2743 597.4680 0.2235 2
|
| 37 |
+
|
| 38 |
+
--- tripwire checks ---
|
| 39 |
+
N3_vqce_nobonus_nat vs N1_vqce_nat: final gap=0.0086 -> UNRESOLVED (< 2*sigma)
|
| 40 |
+
N1_vqce_nat vs N0_raw_nat: final gap=0.0226 -> RESOLVED
|
| 41 |
+
N0_raw_nat vs N2_vqce_multi_nat: final gap=-0.0136 -> UNRESOLVED (< 2*sigma)
|
| 42 |
+
N0_raw_nat: TRANSFER INVERSION — best at ckpt #2 (0.1938) vs final (0.2061); synthetic overfit persists
|
| 43 |
+
N2_vqce_multi_nat: TRANSFER INVERSION — best at ckpt #1 (0.1913) vs final (0.1925); synthetic overfit persists
|
| 44 |
+
|
| 45 |
+
```
|