Upload benchmarks/gemma_baseline_summary.md with huggingface_hub

benchmarks/gemma_baseline_summary.md

@@ -0,0 +1,98 @@
+# Gemma Baseline: OIM v0 vs the Actual Base-Model Tokenizer
+
+Date: 2026-06-27
+Status: decision-relevant baseline added. Closes the "compare Gemma" gap that
+earlier pilots left open because Gemma was gated.
+
+## Why This Matters
+
+Every prior tokenizer audit compared OIM against XLM-R, GPT-2, and cl100k, but
+**not** against Gemma — the tokenizer our own models (`google/gemma-4-E2B-it`)
+actually ship with. That made the central decision ("is an OIM tokenizer worth
+it for our Gemma fine-tunes?") unanswerable.
+
+The Gemma tokenizer turned out to be loadable **without gated access** via the
+non-gated Unsloth mirror `unsloth/gemma-4-E2B-it` (vocab 262,144 — identical to
+our base). So we can now benchmark against the real baseline, no token required.
+
+## Method
+
+- Tokenizers: `unsloth/gemma-4-E2B-it` (our base), `tiktoken/cl100k_base`,
+  OIM `oim-script-equitable-regional-sp-bpe-32k-native-weighted-v0`.
+- Held-out sets (same files used for the v0 audit):
+  - mixed: `script_equitable_v0_heldout.jsonl` (10,000 rows)
+  - Latin: `script_equitable_v0_latn_heldout.jsonl` (9,500 rows)
+  - native: derived from the mixed heldout, `script != Latn` (500 rows across
+    jawi, sunda, jawa, lampung, batak, bali, pegon, lontara)
+- Metric: tokens/word, chars/token, fragmented-word rate over identical text.
+
+## Result
+
+| slice | tokenizer | tokens/word | chars/token | fragmented words |
+|---|---|---:|---:|---:|
+| mixed | gemma-4-E2B-it | 2.026 | 3.439 | 71.5% |
+| mixed | cl100k_base | 2.409 | 2.892 | 80.5% |
+| mixed | **OIM v0** | **1.526** | 4.567 | **22.4%** |
+| Latin | gemma-4-E2B-it | 1.992 | 3.515 | 71.5% |
+| Latin | cl100k_base | 2.377 | 2.946 | 80.4% |
+| Latin | **OIM v0** | **1.527** | 4.585 | **22.2%** |
+| native | gemma-4-E2B-it | 5.785 | 0.501 | 78.0% |
+| native | cl100k_base | 5.972 | 0.485 | 82.6% |
+| native | **OIM v0** | **1.313** | 2.206 | **36.9%** |
+
+OIM reduction vs Gemma in tokens/word:
+
+- mixed: **−24.7%** (2.026 → 1.526)
+- Latin: **−23.3%** (1.992 → 1.527)
+- native aksara: **−77.3%, i.e. 4.4× fewer tokens** (5.785 → 1.313)
+
+## Interpretation
+
+- **Gemma byte-fragments native aksara.** chars/token = 0.50 means Gemma emits
+  ~2 tokens per character on Bali/Jawa/Sunda/Lontara/etc. — the Bengali-BPE
+  byte-fallback failure mode, now confirmed for our exact base model on
+  Indonesian scripts. cl100k is no better (0.49). OIM keeps real Unicode-aware
+  aksara sub-words (chars/token 2.21).
+- **Even Latin regional text pays a Gemma tax.** OIM saves ~23% tokens/word and
+  cuts fragmented words from 71.5% to ~22% on Latin-script Indonesian/regional
+  text — not just aksara.
+- **Decision gate cleared.** The strategy doc's threshold was "≥25% tokens/word
+  reduction vs Gemma/XLM-R on held-out native script, plus downstream CPT win."
+  The native-script reduction vs Gemma is **77%**, far past the 25% gate. The
+  remaining gate condition is the downstream CPT validation win, which needs a
+  vocabulary-extension + CPT experiment (GPU).
+
+## Caveats
+
+- The native slice is 500 rows and mixes Brahmic (bali, jawa, sunda, lontara,
+  batak), Arabic-script (jawi, pegon), and Lampung proxy/Latin notation. The
+  aggregate native number is dominated by whichever scripts have the most rows;
+  per-script breakdown lives in
+  `tokenizer_benchmark_script_equitable_v0_by_native_script.md`.
+- tokens/word for SentencePiece on aksara can be distorted by whitespace-based
+  word counting; the relative comparison on identical text is the valid signal,
+  and fragmented-word rate corroborates it.
+- This measures compression only. It does **not** yet prove a modeling win —
+  that requires vocab-extension + CPT and a validation-loss comparison.
+
+## Reproduce
+
+```bash
+OIM=artifacts/tokenizer_hf_preview/oim-script-equitable-regional-sp-bpe-32k-native-weighted-v0/oim-script-equitable-regional-v0-native-weighted-32k-sp-bpe.model
+uv run python scripts/tokenizer_benchmark.py \
+  --input artifacts/tokenizer_corpora/scale_v0/script_equitable_v0_heldout.jsonl \
+  --language regional-mixed --field text \
+  --tokenizer unsloth/gemma-4-E2B-it \
+  --sentencepiece-model "$OIM" --tiktoken cl100k_base \
+  --output-md docs/data_audits/tokenizer_benchmark_gemma_vs_oim_mixed.md
+# repeat with the latin and native-derived heldout files
+```
+
+## Next
+
+1. Vocabulary-extension experiment: add OIM's high-value aksara tokens to the
+   Gemma vocab, initialize embeddings, run short CPT, measure validation loss.
+   This converts the compression win into a modeling claim.
+2. Verify / re-upload the private HF artifact (blocked on a local HF token).
+3. Per-script native heldouts with larger row counts once more native corpora
+   are sourced.