---
library_name: tokenizers
language:
- id
- jv
- su
- ban
- bug
- ljp
tags:
- tokenizer
- indonesian
- regional-languages
- low-resource-languages
- sentencepiece
- script-equitable
- native-script
- aksara
- native-weighted
license: other
---

# OIM Script-Equitable Regional SentencePiece BPE 48k (Native-Weighted v1)

A 48,000-token Unicode-aware SentencePiece BPE tokenizer for Indonesian and its
regional languages, trained with **fertility-aware native-script weighting** so
aksara (Bali, Jawa, Sunda, Lontara, and related scripts) are not byte-fragmented
the way general-purpose tokenizers fragment them.

Part of **Open Indonesia Models (OIM)**. This is the stronger successor to
[`oim-script-equitable-regional-sp-bpe-32k-native-weighted-v0`](https://huggingface.co/timothydillan/oim-script-equitable-regional-sp-bpe-32k-native-weighted-v0):
a ~3.85× larger corpus (504k rows), a 48k vocab, and a stronger native upweight
(30×) that keeps the native-script corpus share at ~32% despite the larger Latin
pool. Research/audit artifact and a candidate base for tokenizer surgery — not a
drop-in LLM tokenizer replacement without (continued) pretraining.

## Headline: vs the base model we actually use

Our regional LLMs fine-tune `google/gemma-4-E2B-it`. Measured on a held-out mix
of Indonesian/regional Latin and native-script text, v1 vs Gemma's own tokenizer
(262k vocab):

| held-out slice | Gemma-4-E2B-it | cl100k_base | OIM v0 (32k) | **OIM v1 (48k)** | v1 vs Gemma |
|---|---:|---:|---:|---:|---:|
| mixed, tokens/word | 2.026 | 2.407 | 1.539 | **1.464** | **−27.7%** |
| Latin, tokens/word | 1.990 | 2.373 | 1.541 | **1.467** | **−26.3%** |
| **native aksara, tokens/word** | 5.785 | 5.972 | 1.313 | **1.191** | **−79.4% (4.9×)** |
| native, chars/token | 0.50 | 0.49 | 2.21 | **2.43** | — |
| native, fragmented words | 78.0% | 82.6% | 36.9% | **30.5%** | — |

Gemma (like cl100k) emits ~2 tokens per character on native aksara — the
byte-fallback failure mode. v1 keeps real Unicode-aware aksara sub-words and cuts
native-script token count **4.9×**.

## What changed vs v0

Evaluated on v1's held-out set (v1 trained leak-free against it):

| slice | OIM v0 (32k) | OIM v1 (48k) | change |
|---|---:|---:|---:|
| mixed tokens/word | 1.539 | **1.464** | **−4.9%** |
| Latin tokens/word | 1.541 | **1.467** | **−4.8%** |
| native tokens/word | 1.313 | **1.191** | **−9.3%** |
| mixed fragmented words | 22.9% | **18.6%** | −4.3 pts |
| native fragmented words | 36.9% | **30.5%** | −6.4 pts |

The larger Latin pool did **not** regress native script — the stronger upweight
held the native share, and native improved most.

## Corpus

504,112-row native-weighted corpus:

- CC-100 Indonesian / Javanese / Sundanese Latin — 90k train rows each
- Wikipedia Indonesian / Javanese / Sundanese (10k each) + Balinese (15k)
- NusaAksara native-script transcriptions — upweighted **30×**
- NusaAksara Latin transliterations — upweighted **4×**
- Balinese research SFT message content — 8k rows

Native-script rows are ~31.9% of the weighted training corpus. Vocab 48,000,
character coverage 1.0 (every aksara codepoint stays atomic; no byte fallback for
covered scripts).

## Intended use

- tokenizer fertility / "token tax" audits by language and script;
- retrieval and chunking for Indonesian regional-language corpora;
- vocabulary-extension or from-scratch regional-LM tokenizer research.

Not a drop-in LLM tokenizer replacement without (continued) pretraining.

## Provenance & licensing

Ships **derived tokenizer model files** plus benchmark JSON and a corpus
manifest. Does **not** redistribute raw corpus rows.

- **CC-100** (`data.statmt.org/cc-100`) — derived from Common Crawl; Common Crawl terms.
- **Wikipedia** (`wikimedia/wikipedia`) — CC-BY-SA 4.0; attribution to Wikipedia contributors.
- **NusaAksara** (`NusaAksara/NusaAksara`, ACL 2025) — see dataset card for license.
- **Balinese research SFT** — OIM project data (message content only).

`license: other` reflects mixed provenance. Honor each source's
attribution/share-alike terms; to assign one SPDX license to the model files
alone, treat them as CC-BY-SA-4.0 to respect the Wikipedia share-alike input.

## Files

- `oim-script-equitable-regional-v1-native-weighted-48k-sp-bpe.model` — SentencePiece model
- `oim-script-equitable-regional-v1-native-weighted-48k-sp-bpe.vocab` — vocabulary
- `MANIFEST.json` — file hashes and metadata
- `benchmarks/*.json` — held-out comparison vs Gemma / cl100k / v0
- `provenance/*.json` — corpus build manifest (sources, weights, row counts)

## Usage

```python
import sentencepiece as spm
sp = spm.SentencePieceProcessor(
    model_file="oim-script-equitable-regional-v1-native-weighted-48k-sp-bpe.model"
)
print(sp.encode("Om Swastiastu, titiang saking Bali.", out_type=str))
```

## Status

Public research v1. The compression win vs Gemma is established; the downstream
modeling win (vocabulary extension + continued pretraining, measured by
validation loss) is the next step and is not yet claimed here.