Indic BPE Tokenizer
Experimental Hindi–Hinglish BPE tokenizer research project focused on native Hindi, Romanized Hindi, Hinglish code-mixed text, mixed-script text, and informal chat.
The current release candidate is a 32K Metaspace BPE tokenizer with byte fallback for reversible open-vocabulary handling.
Current status
release track: v0.1-experimental
tokenizer mode: metaspace_bpe_byte_fallback
vocabulary size: 32,000
selected artifact: hindi_metaspace_bpe_32k_balanced_plus_dakshina3x_plus_synthetic_romanized_50k
unknown-token rate: 0.0000%
round-trip failures: 0
This is a research release candidate, not a production-certified commercial tokenizer.
Why this project exists
Standard multilingual tokenizers often work reasonably well for many languages, but they are not optimized for how Hindi is commonly written online.
Real usage often mixes:
native Devanagari Hindi
Romanized Hindi
English words
Hinglish code-mixing
emojis
informal spelling variants
mixed Devanagari-Latin text
This project explores whether a focused tokenizer can improve compression and robustness for these patterns while preserving exact encode/decode recovery.
Latest benchmark results
Candidate artifact:
artifacts/hindi_metaspace_bpe_32k_balanced_plus_dakshina3x_plus_synthetic_romanized_50k
Benchmark suite:
large
| Domain | Sentences | Candidate tokens / word | XLM-R tokens / word | Compression gain vs XLM-R |
|---|---|---|---|---|
| Native Hindi | 8,108 | 1.2955 | 1.5820 | +18.11% |
| Romanized Hindi | 1,000 | 3.1570 | 2.7750 | -13.77% |
| Hinglish code-mixed | 1,000 | 1.3939 | 1.4363 | +2.95% |
| Mixed script | 1,000 | 1.4214 | 1.5257 | +6.84% |
| Informal chat | 1,000 | 1.3223 | 1.4524 | +8.96% |
Core robustness checks:
unknown-token rate: 0.0000%
round-trip failures: 0
Romanized Hindi remains the main improvement area, but the latest experiment improved it materially from the earlier -19.53% baseline to -13.77%.
Training data lineage
The selected release candidate was trained on a balanced mix of:
filtered Hindi Wikipedia training split
+
Aksharantar Hindi Romanized shard: 250K lines
+
L3Cube Hinglish cleaned shard: 500K lines
+
Dakshina Hindi Romanized sentences weighted 3x
+
synthetic Romanized-Hindi sentences generated from the Hindi Wikipedia training split
The synthetic Romanized-Hindi shard is derived only from training data. Held-out evaluation data is not used for synthetic augmentation.
Repository structure
indic_bpe_tokenizer/ Python package
scripts/ Corpus-building and release-freezing scripts
tests/ Unit and release-load tests
data/ Local training and evaluation data; usually not committed
artifacts/ Trained tokenizer artifacts; usually not committed
reports/ Benchmark and corpus-stat reports
release/ Hugging Face release staging files
Experiment_Milestones.md Experiment log and milestone decisions
Possible_Extensions.md Future project directions
Installation
Create and activate the main tokenizer environment:
python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -e .
For synthetic Romanized-Hindi generation, use the separate IndicXlit environment documented in the milestone notes. Do not install the old IndicXlit dependency stack into the main tokenizer environment.
Train a tokenizer
Example training command for the current release candidate:
python -m indic_bpe_tokenizer.train_tokenizer \
--data-dir data/train/experiments/balanced_plus_dakshina3x_plus_synthetic_romanized_50k \
--artifact-dir artifacts/hindi_metaspace_bpe_32k_balanced_plus_dakshina3x_plus_synthetic_romanized_50k \
--tokenizer-mode metaspace_bpe_byte_fallback
Run the benchmark suite
python -m indic_bpe_tokenizer.benchmark_suite \
--candidate-artifact artifacts/hindi_metaspace_bpe_32k_balanced_plus_dakshina3x_plus_synthetic_romanized_50k/tokenizer.json \
--suite large
Freeze a Hugging Face release package
python scripts/freeze_tokenizer_release.py \
--artifact-dir artifacts/hindi_metaspace_bpe_32k_balanced_plus_dakshina3x_plus_synthetic_romanized_50k \
--output-dir release/hindi-hinglish-bpe-tokenizer-v0.1-experimental-frozen \
--suite large \
--overwrite
The frozen release package should contain:
README.md
DATASET_LICENSE_NOTICE.md
experiment_metadata.json
tokenizer.json
vocab.json
merges.txt
benchmark_reports/
release_manifest.json
SHA256SUMS.txt
Test the frozen release
TOKENIZER_RELEASE_DIR=release/hindi-hinglish-bpe-tokenizer-v0.1-experimental-frozen \
pytest -v tests/test_release_load.py
Expected result after a successful freeze:
7 passed
Load the tokenizer
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")
text = "mera claim अभी तक process nahi hua 😊"
encoding = tokenizer.encode(text)
decoded = tokenizer.decode(encoding.ids, skip_special_tokens=False)
assert decoded == text
Licensing and dataset-use warning
This project uses third-party datasets with different license conditions. In particular, the current training mix includes L3Cube HingCorpus, which has non-commercial and share-alike restrictions.
Treat this tokenizer artifact as a research and non-commercial experimental release unless a separate legal review confirms broader reuse rights.
See:
DATASET_LICENSE_NOTICE.md
Release positioning
Use this artifact as:
v0.1-experimental
V1 research release candidate
Do not describe it as:
v1.0 production tokenizer
unrestricted commercial tokenizer
trained language model
embedding model
chat model
Next improvement areas
Future work should focus on:
stronger real Romanized-Hindi sentence corpus
natural mixed-script evaluation set
commercially cleaner training corpus
synthetic Romanized-Hindi 250K ablation
additional Indic-language expansion
tokenizer integration with embedding and transformer experiments
Project milestone summary
See Experiment_Milestones.md for the full experiment history, ablations, acceptance criteria, and release decisions.