---
license: apache-2.0
tags: [time-series, forecasting, foundation-model, tokenizer-ablation, toto-2-recipe]
---
# TSFM tokenizer ablation — Toto-2-4m-recipe clone (v10)

VQ-VAE OPTIMIZATION ablation (codebook health + next-code objective vs raw patches) on a ~3.6M-param decoder-only patched transformer
trained on the OFFICIAL TempoPFN synthetic prior generators (cloned from
automl/TempoPFN; GP/KernelSynth families capped at 6,144 steps, so the long
pool uses the cheap families only) with contiguous patch
masking, a 9-level quantile head, NorMuon+AdamW, and index RoPE at
time-scaled positions. Horizon decoding is fixed patch-32 in every arm; only
the CONTEXT tokenizer varies:

| arm | tokenizer | history | ctx tokens |
|-----|-----------|---------|-----------|
| T0  | fixed-32 (control) | 4,096 | 128 |
| T1  | pyramid, iso-context | 4,096 | 44 |
| T2  | pyramid, iso-token | 16,384 | 128 |
| T3  | adaptive equal-surprise | 16,384 | 128 |

Each subfolder is one (arm, seed) run: `model.pt` (final), `ckpt_15000.pt`
(rank-stability snapshot), `config.json`, `results.json` (dev-GIFT CRPS +
long-horizon probe). Dev metrics use a fixed 14-task GIFT-Eval subset — NOT
the full leaderboard; treat numbers as ablation-internal, not comparable to
published GIFT scores. Generated by the v10 experiment notebook.

## Results
```
=== transfer curve: GM-CRPS by checkpoint (dev-14, mean over seeds) ===
                     gm_avg2  gm_avg2_std  gm@7500  gm@15000  gm@22500  gm@final  gm_final_std  long_season  params_m       perp  probe_amp  n
arm                                                                                                                                           
R3_k4096_nat          0.1795          NaN   0.2066    0.1889    0.1840    0.1750           NaN       0.2512    4.2743  1597.8119     0.2750  1
R4_softce_nat         0.1817       0.0048   0.2018    0.1960    0.1834    0.1800        0.0099       0.2508    3.8780   558.6021     0.2380  2
N3_vqce_nobonus_nat   0.1835       0.0021   0.1987    0.1881    0.1883    0.1787        0.0034       0.2485    3.8780   555.6374     0.1750  3
R1_w01_nat            0.1836          NaN   0.1908    0.1854    0.1862    0.1809           NaN       0.2538    3.8780   570.1680     0.2180  1
R2_k2048_nat          0.1866       0.0012   0.2137    0.1935    0.1898    0.1834        0.0048       0.2542    4.0101   917.4004     0.2910  2
R5_horizonce_nat      0.1873       0.0015   0.2021    0.1873    0.1872    0.1873        0.0031       0.2519    4.0430   570.5395     0.1965  2
N1_vqce_nat           0.1909       0.0109   0.2121    0.2052    0.1983    0.1835        0.0049       0.2550    3.8780   620.2301     0.2280  3
N5_vqce_w05_nat       0.1959          NaN   0.2092    0.2005    0.1968    0.1949           NaN       0.2593    3.8780   598.1758     0.2410  1
N0_raw_nat            0.2037       0.0042   0.2011    0.1938    0.2014    0.2061        0.0048       0.2635    3.5802        NaN     0.2253  3
N2_vqce_multi_nat     0.2039       0.0189   0.1913    0.1943    0.2154    0.1925        0.0034       0.2598    4.2743   597.4680     0.2235  2
N4_rawce_nat          0.2404       0.0030   0.2043    0.2293    0.2371    0.2437        0.0028       0.2832    3.7452   461.1100     0.3755  2

--- tripwire checks ---
  R3_k4096_nat vs R4_softce_nat: final gap=0.0050 -> UNRESOLVED (< 2*sigma)
  R4_softce_nat vs N3_vqce_nobonus_nat: final gap=-0.0013 -> UNRESOLVED (< 2*sigma)
  N3_vqce_nobonus_nat vs R1_w01_nat: final gap=0.0022 -> UNRESOLVED (< 2*sigma)
  R1_w01_nat vs R2_k2048_nat: final gap=0.0025 -> UNRESOLVED (< 2*sigma)
  R2_k2048_nat vs R5_horizonce_nat: final gap=0.0039 -> UNRESOLVED (< 2*sigma)
  R5_horizonce_nat vs N1_vqce_nat: final gap=-0.0038 -> UNRESOLVED (< 2*sigma)
  N1_vqce_nat vs N5_vqce_w05_nat: final gap=0.0114 -> UNRESOLVED (< 2*sigma)
  N5_vqce_w05_nat vs N0_raw_nat: final gap=0.0111 -> UNRESOLVED (< 2*sigma)
  N0_raw_nat vs N2_vqce_multi_nat: final gap=-0.0136 -> UNRESOLVED (< 2*sigma)
  N2_vqce_multi_nat vs N4_rawce_nat: final gap=0.0513 -> RESOLVED
  R5_horizonce_nat: TRANSFER INVERSION — best at ckpt #3 (0.1872) vs final (0.1873); synthetic overfit persists
  N0_raw_nat: TRANSFER INVERSION — best at ckpt #2 (0.1938) vs final (0.2061); synthetic overfit persists
  N2_vqce_multi_nat: TRANSFER INVERSION — best at ckpt #1 (0.1913) vs final (0.1925); synthetic overfit persists
  N4_rawce_nat: TRANSFER INVERSION — best at ckpt #1 (0.2043) vs final (0.2437); synthetic overfit persists

```