timothydillan/oim-script-equitable-regional-sp-bpe-48k-native-weighted-v1

OIM Script-Equitable Regional SentencePiece BPE 48k (Native-Weighted v1)

A 48,000-token Unicode-aware SentencePiece BPE tokenizer for Indonesian and its regional languages, trained with fertility-aware native-script weighting so aksara (Bali, Jawa, Sunda, Lontara, and related scripts) are not byte-fragmented the way general-purpose tokenizers fragment them.

Part of Open Indonesia Models (OIM). This is the stronger successor to oim-script-equitable-regional-sp-bpe-32k-native-weighted-v0: a ~3.85× larger corpus (504k rows), a 48k vocab, and a stronger native upweight (30×) that keeps the native-script corpus share at ~32% despite the larger Latin pool. Research/audit artifact and a candidate base for tokenizer surgery — not a drop-in LLM tokenizer replacement without (continued) pretraining.

Headline: vs the base model we actually use

Our regional LLMs fine-tune google/gemma-4-E2B-it. Measured on a held-out mix of Indonesian/regional Latin and native-script text, v1 vs Gemma's own tokenizer (262k vocab):

held-out slice	Gemma-4-E2B-it	cl100k_base	OIM v0 (32k)	OIM v1 (48k)	v1 vs Gemma
mixed, tokens/word	2.026	2.407	1.539	1.464	−27.7%
Latin, tokens/word	1.990	2.373	1.541	1.467	−26.3%
native aksara, tokens/word	5.785	5.972	1.313	1.191	−79.4% (4.9×)
native, chars/token	0.50	0.49	2.21	2.43	—
native, fragmented words	78.0%	82.6%	36.9%	30.5%	—

Gemma (like cl100k) emits ~2 tokens per character on native aksara — the byte-fallback failure mode. v1 keeps real Unicode-aware aksara sub-words and cuts native-script token count 4.9×.

What changed vs v0

Evaluated on v1's held-out set (v1 trained leak-free against it):

slice	OIM v0 (32k)	OIM v1 (48k)	change
mixed tokens/word	1.539	1.464	−4.9%
Latin tokens/word	1.541	1.467	−4.8%
native tokens/word	1.313	1.191	−9.3%
mixed fragmented words	22.9%	18.6%	−4.3 pts
native fragmented words	36.9%	30.5%	−6.4 pts

The larger Latin pool did not regress native script — the stronger upweight held the native share, and native improved most.

Corpus

504,112-row native-weighted corpus:

• CC-100 Indonesian / Javanese / Sundanese Latin — 90k train rows each
• Wikipedia Indonesian / Javanese / Sundanese (10k each) + Balinese (15k)
• NusaAksara native-script transcriptions — upweighted 30×
• NusaAksara Latin transliterations — upweighted 4×
• Balinese research SFT message content — 8k rows

Native-script rows are ~31.9% of the weighted training corpus. Vocab 48,000, character coverage 1.0 (every aksara codepoint stays atomic; no byte fallback for covered scripts).

Intended use

• tokenizer fertility / "token tax" audits by language and script;
• retrieval and chunking for Indonesian regional-language corpora;
• vocabulary-extension or from-scratch regional-LM tokenizer research.

Not a drop-in LLM tokenizer replacement without (continued) pretraining.

Provenance & licensing

Ships derived tokenizer model files plus benchmark JSON and a corpus manifest. Does not redistribute raw corpus rows.

• CC-100 (data.statmt.org/cc-100) — derived from Common Crawl; Common Crawl terms.
• Wikipedia (wikimedia/wikipedia) — CC-BY-SA 4.0; attribution to Wikipedia contributors.
• NusaAksara (NusaAksara/NusaAksara, ACL 2025) — see dataset card for license.
• Balinese research SFT — OIM project data (message content only).

license: other reflects mixed provenance. Honor each source's attribution/share-alike terms; to assign one SPDX license to the model files alone, treat them as CC-BY-SA-4.0 to respect the Wikipedia share-alike input.

Files

• oim-script-equitable-regional-v1-native-weighted-48k-sp-bpe.model — SentencePiece model
• oim-script-equitable-regional-v1-native-weighted-48k-sp-bpe.vocab — vocabulary
• MANIFEST.json — file hashes and metadata
• benchmarks/*.json — held-out comparison vs Gemma / cl100k / v0
• provenance/*.json — corpus build manifest (sources, weights, row counts)

Usage

import sentencepiece as spm
sp = spm.SentencePieceProcessor(
    model_file="oim-script-equitable-regional-v1-native-weighted-48k-sp-bpe.model"
)
print(sp.encode("Om Swastiastu, titiang saking Bali.", out_type=str))

Status

Public research v1. The compression win vs Gemma is established; the downstream modeling win (vocabulary extension + continued pretraining, measured by validation loss) is the next step and is not yet claimed here.