timothydillan/oim-script-equitable-regional-sp-bpe-32k-native-weighted-v0

OIM Script-Equitable Regional SentencePiece BPE 32k (Native-Weighted v0)

A 32,000-token Unicode-aware SentencePiece BPE tokenizer for Indonesian and its regional languages, trained with fertility-aware native-script weighting so that aksara (Bali, Jawa, Sunda, Lontara, and related scripts) are not byte- fragmented the way general-purpose tokenizers fragment them.

Part of the Open Indonesia Models (OIM) project. This is a research/audit artifact and a candidate base for tokenizer surgery (vocabulary extension / from-scratch regional LMs) — not a drop-in replacement for an existing pretrained LLM tokenizer unless that model is (continued-)pretrained with it.

Headline: vs the base model we actually use

Our regional LLMs fine-tune google/gemma-4-E2B-it. Measured on a held-out mix of Indonesian/regional Latin and native-script text, OIM v0 vs Gemma's own tokenizer (262k vocab):

held-out slice	Gemma-4-E2B-it	cl100k_base	OIM v0	OIM vs Gemma
mixed, tokens/word	2.026	2.409	1.526	−24.7%
Latin, tokens/word	1.992	2.377	1.527	−23.3%
native aksara, tokens/word	5.785	5.972	1.313	−77.3% (4.4×)
native, chars/token	0.50	0.49	2.21	—

Gemma (like cl100k) emits roughly two tokens per character on native aksara — the byte-fallback failure mode. OIM keeps real Unicode-aware aksara sub-words, cutting native-script token count 4.4× and clearing the project's "≥25% reduction vs base tokenizer on native script" decision gate.

Corpus

Trained on a 130,969-row native-weighted research corpus:

• CC-100 Indonesian / Javanese / Sundanese Latin samples
• Wikipedia Indonesian / Balinese / Javanese / Sundanese samples
• NusaAksara native-script transcriptions
• NusaAksara Latin transliterations
• Balinese research SFT message content

Native NusaAksara rows are upweighted 8× and transliterations 2× so that high-frequency Indonesian/English Latin does not dominate merge selection. The unweighted 32k variant scored better on Latin but regressed on native script (51.7% → this model's 36.9% native fragmented-word rate); native weighting is the better script-equitable trade.

Metrics

Lower tokens/word and fragmented words are better. Held-out comparison against XLM-R, cl100k, and earlier OIM variants:

language	tokenizer	tokens/word	chars/token	fragmented words
script-equitable-v0-heldout	xlm-roberta-base	1.797	3.878	42.6%
script-equitable-v0-heldout	tiktoken/cl100k_base	2.409	2.892	80.5%
script-equitable-v0-heldout	oim-v0-32k (unweighted)	1.513	4.603	21.6%
script-equitable-v0-heldout	oim-v0-32k native-weighted	1.526	4.567	22.4%
script-equitable-v0-latn-heldout	xlm-roberta-base	1.803	3.883	42.3%
script-equitable-v0-latn-heldout	tiktoken/cl100k_base	2.377	2.946	80.4%
script-equitable-v0-latn-heldout	oim-v0-32k (unweighted)	1.511	4.636	21.3%
script-equitable-v0-latn-heldout	oim-v0-32k native-weighted	1.527	4.585	22.2%
script-equitable-v0-native-heldout	xlm-roberta-base	1.028*	2.816	79.0%
script-equitable-v0-native-heldout	tiktoken/cl100k_base	5.972	0.485	82.6%
script-equitable-v0-native-heldout	oim-v0-32k (unweighted)	1.829	1.583	51.7%
script-equitable-v0-native-heldout	oim-v0-32k native-weighted	1.313	2.206	36.9%

* XLM-R's low native tokens/word is an artifact of whitespace-based word counting on scripts it largely maps to <unk>; read it alongside its 79% fragmented-word rate. Per-script native breakdown (bali, batak, jawa, jawi, lampung, lontara, pegon, sunda) is in benchmarks/.

Note: Lampung rows in NusaAksara are currently proxy/Latin notation rather than encoded Lampung script, so Lampung is not yet a real native-script benchmark.

Intended use

• tokenizer fertility / "token tax" audits by language and script;
• retrieval and chunking for Indonesian regional-language corpora;
• vocabulary-extension or from-scratch regional-LM tokenizer research.

Not a drop-in LLM tokenizer replacement without (continued) pretraining.

Provenance & licensing

This repository ships derived tokenizer model files (a SentencePiece model and its vocabulary) plus benchmark JSON and a corpus manifest. It does not redistribute raw corpus rows.

Source corpora and their terms:

• CC-100 (data.statmt.org/cc-100) — derived from Common Crawl; subject to Common Crawl terms of use.
• Wikipedia (wikimedia/wikipedia) — CC-BY-SA 4.0; attribution to Wikipedia contributors.
• NusaAksara (NusaAksara/NusaAksara, ACL 2025) — see dataset card for license; used for native-script transcriptions and transliterations.
• Balinese research SFT — OIM project data (message content only).

license: other reflects this mixed provenance. The artifact is a statistical tokenizer derived from the above; downstream users should honor each source's attribution/share-alike terms. If you need a single SPDX license for the model files alone, treat them as CC-BY-SA-4.0 to respect the Wikipedia share-alike input.

Files

• oim-script-equitable-regional-v0-native-weighted-32k-sp-bpe.model — SentencePiece model
• oim-script-equitable-regional-v0-native-weighted-32k-sp-bpe.vocab — vocabulary
• MANIFEST.json — file hashes and metadata
• benchmarks/*.json — held-out and per-script benchmark results
• provenance/*.json — corpus build manifest (sources, weights, row counts)

Usage

import sentencepiece as spm
sp = spm.SentencePieceProcessor(
    model_file="oim-script-equitable-regional-v0-native-weighted-32k-sp-bpe.model"
)
print(sp.encode("Om Swastiastu, titiang saking Bali.", out_type=str))

Status

Public research v0. The compression win vs Gemma is established; the downstream modeling win (vocabulary extension + continued pretraining, measured by validation loss) is the next step and is not yet claimed here. A larger corpus v1 is in progress.