---
library_name: tokenizers
language:
- id
- jv
- su
- ban
- bug
- ljp
tags:
- tokenizer
- indonesian
- regional-languages
- low-resource-languages
- sentencepiece
- script-equitable
- native-script
- aksara
- native-weighted
license: other
---

# OIM Script-Equitable Regional SentencePiece BPE 32k (Native-Weighted v0)

A 32,000-token Unicode-aware SentencePiece BPE tokenizer for Indonesian and its
regional languages, trained with **fertility-aware native-script weighting** so
that aksara (Bali, Jawa, Sunda, Lontara, and related scripts) are not byte-
fragmented the way general-purpose tokenizers fragment them.

Part of the **Open Indonesia Models (OIM)** project. This is a research/audit
artifact and a candidate base for tokenizer surgery (vocabulary extension /
from-scratch regional LMs) — not a drop-in replacement for an existing
pretrained LLM tokenizer unless that model is (continued-)pretrained with it.

## Headline: vs the base model we actually use

Our regional LLMs fine-tune `google/gemma-4-E2B-it`. Measured on a held-out
mix of Indonesian/regional Latin and native-script text, OIM v0 vs Gemma's own
tokenizer (262k vocab):

| held-out slice | Gemma-4-E2B-it | cl100k_base | **OIM v0** | OIM vs Gemma |
|---|---:|---:|---:|---:|
| mixed, tokens/word | 2.026 | 2.409 | **1.526** | **−24.7%** |
| Latin, tokens/word | 1.992 | 2.377 | **1.527** | **−23.3%** |
| **native aksara, tokens/word** | 5.785 | 5.972 | **1.313** | **−77.3% (4.4×)** |
| native, chars/token | 0.50 | 0.49 | **2.21** | — |

Gemma (like cl100k) emits roughly **two tokens per character** on native aksara
— the byte-fallback failure mode. OIM keeps real Unicode-aware aksara
sub-words, cutting native-script token count **4.4×** and clearing the project's
"≥25% reduction vs base tokenizer on native script" decision gate.

## Corpus

Trained on a 130,969-row native-weighted research corpus:

- CC-100 Indonesian / Javanese / Sundanese Latin samples
- Wikipedia Indonesian / Balinese / Javanese / Sundanese samples
- NusaAksara native-script transcriptions
- NusaAksara Latin transliterations
- Balinese research SFT message content

Native NusaAksara rows are upweighted **8×** and transliterations **2×** so that
high-frequency Indonesian/English Latin does not dominate merge selection. The
unweighted 32k variant scored better on Latin but regressed on native script
(51.7% → this model's 36.9% native fragmented-word rate); native weighting is the
better script-equitable trade.

## Metrics

Lower `tokens/word` and `fragmented words` are better. Held-out comparison
against XLM-R, cl100k, and earlier OIM variants:

| language | tokenizer | tokens/word | chars/token | fragmented words |
|---|---:|---:|---:|---:|
| script-equitable-v0-heldout | xlm-roberta-base | 1.797 | 3.878 | 42.6% |
| script-equitable-v0-heldout | tiktoken/cl100k_base | 2.409 | 2.892 | 80.5% |
| script-equitable-v0-heldout | oim-v0-32k (unweighted) | 1.513 | 4.603 | 21.6% |
| script-equitable-v0-heldout | **oim-v0-32k native-weighted** | 1.526 | 4.567 | 22.4% |
| script-equitable-v0-latn-heldout | xlm-roberta-base | 1.803 | 3.883 | 42.3% |
| script-equitable-v0-latn-heldout | tiktoken/cl100k_base | 2.377 | 2.946 | 80.4% |
| script-equitable-v0-latn-heldout | oim-v0-32k (unweighted) | 1.511 | 4.636 | 21.3% |
| script-equitable-v0-latn-heldout | **oim-v0-32k native-weighted** | 1.527 | 4.585 | 22.2% |
| script-equitable-v0-native-heldout | xlm-roberta-base | 1.028* | 2.816 | 79.0% |
| script-equitable-v0-native-heldout | tiktoken/cl100k_base | 5.972 | 0.485 | 82.6% |
| script-equitable-v0-native-heldout | oim-v0-32k (unweighted) | 1.829 | 1.583 | 51.7% |
| script-equitable-v0-native-heldout | **oim-v0-32k native-weighted** | 1.313 | 2.206 | 36.9% |

\* XLM-R's low native tokens/word is an artifact of whitespace-based word
counting on scripts it largely maps to `<unk>`; read it alongside its 79%
fragmented-word rate. Per-script native breakdown (bali, batak, jawa, jawi,
lampung, lontara, pegon, sunda) is in `benchmarks/`.

Note: Lampung rows in NusaAksara are currently proxy/Latin notation rather than
encoded Lampung script, so Lampung is not yet a real native-script benchmark.

## Intended use

- tokenizer fertility / "token tax" audits by language and script;
- retrieval and chunking for Indonesian regional-language corpora;
- vocabulary-extension or from-scratch regional-LM tokenizer research.

Not a drop-in LLM tokenizer replacement without (continued) pretraining.

## Provenance & licensing

This repository ships **derived tokenizer model files** (a SentencePiece model
and its vocabulary) plus benchmark JSON and a corpus manifest. It does **not**
redistribute raw corpus rows.

Source corpora and their terms:

- **CC-100** (`data.statmt.org/cc-100`) — derived from Common Crawl; subject to
  Common Crawl terms of use.
- **Wikipedia** (`wikimedia/wikipedia`) — CC-BY-SA 4.0; attribution to Wikipedia
  contributors.
- **NusaAksara** (`NusaAksara/NusaAksara`, ACL 2025) — see dataset card for
  license; used for native-script transcriptions and transliterations.
- **Balinese research SFT** — OIM project data (message content only).

`license: other` reflects this mixed provenance. The artifact is a statistical
tokenizer derived from the above; downstream users should honor each source's
attribution/share-alike terms. If you need a single SPDX license for the model
files alone, treat them as CC-BY-SA-4.0 to respect the Wikipedia share-alike
input.

## Files

- `oim-script-equitable-regional-v0-native-weighted-32k-sp-bpe.model` — SentencePiece model
- `oim-script-equitable-regional-v0-native-weighted-32k-sp-bpe.vocab` — vocabulary
- `MANIFEST.json` — file hashes and metadata
- `benchmarks/*.json` — held-out and per-script benchmark results
- `provenance/*.json` — corpus build manifest (sources, weights, row counts)

## Usage

```python
import sentencepiece as spm
sp = spm.SentencePieceProcessor(
    model_file="oim-script-equitable-regional-v0-native-weighted-32k-sp-bpe.model"
)
print(sp.encode("Om Swastiastu, titiang saking Bali.", out_type=str))
```

## Status

Public research v0. The compression win vs Gemma is established; the downstream
modeling win (vocabulary extension + continued pretraining, measured by
validation loss) is the next step and is not yet claimed here. A larger
corpus v1 is in progress.