---
license: apache-2.0
tags: [time-series, forecasting, foundation-model, tokenizer-ablation, toto-2-recipe]
---
# TSFM tokenizer ablation — Toto-2-4m-recipe clone (v10)

LEARNED-tokenizer ablation (discrete bottlenecks vs raw patches at equal token count) on a ~3.6M-param decoder-only patched transformer
trained on the OFFICIAL TempoPFN synthetic prior generators (cloned from
automl/TempoPFN; GP/KernelSynth families capped at 6,144 steps, so the long
pool uses the cheap families only) with contiguous patch
masking, a 9-level quantile head, NorMuon+AdamW, and index RoPE at
time-scaled positions. Horizon decoding is fixed patch-32 in every arm; only
the CONTEXT tokenizer varies:

| arm | tokenizer | history | ctx tokens |
|-----|-----------|---------|-----------|
| T0  | fixed-32 (control) | 4,096 | 128 |
| T1  | pyramid, iso-context | 4,096 | 44 |
| T2  | pyramid, iso-token | 16,384 | 128 |
| T3  | adaptive equal-surprise | 16,384 | 128 |

Each subfolder is one (arm, seed) run: `model.pt` (final), `ckpt_15000.pt`
(rank-stability snapshot), `config.json`, `results.json` (dev-GIFT CRPS +
long-horizon probe). Dev metrics use a fixed 14-task GIFT-Eval subset — NOT
the full leaderboard; treat numbers as ablation-internal, not comparable to
published GIFT scores. Generated by the v10 experiment notebook.

## Results
```
=== transfer curve: GM-CRPS by checkpoint (dev-14, mean over seeds) ===
               gm@7500  gm@15000  gm@22500  gm@final  gm_final_std  long_season  params_m      perp  probe_amp  n
arm                                                                                                              
L2_vq_tpfn      0.2219    0.2324    0.2314    0.2089        0.0032       0.2594    3.7130  422.0938     0.2360  3
L0_fixed_tpfn   0.2166    0.2258    0.2101    0.2106        0.0085       0.2632    3.5802       NaN     0.1910  3
L1_fsq_tpfn     0.2400    0.2494    0.2500    0.2333        0.0021       0.2733    3.6827    5.4675     0.3427  3

--- tripwire checks ---
  L2_vq_tpfn vs L0_fixed_tpfn: final gap=0.0017 -> UNRESOLVED (< 2*sigma)
  L0_fixed_tpfn vs L1_fsq_tpfn: final gap=0.0226 -> RESOLVED
  L0_fixed_tpfn: TRANSFER INVERSION — best at ckpt #3 (0.2101) vs final (0.2106); synthetic overfit persists

```