README.md

---
library_name: tokenizers
language:
- fin
tags:
- LangMAP
- unsupervised
- tokenizer
---

# Base + Language-Specific LangMAP — mistral-7b-v03 × fin_Latn

Unsupervised tokenization specialised for **fin_Latn**, derived from the
**mistral-7b-v03** base BPE tokenizer using the LangMAP framework.

This repository bundles:
- `base_tokenizer.json` — joint LAS Unigram base
- `langspec_fin_Latn.json` — language-specific overlay (re-EM on fin_Latn corpus)
- `tokenizer.json` — alias for the overlay (default load target)

Inference uses base + language-specific scores together (the LangMAP variant);
do not use the bare overlay or base on its own.

Trained from job `smokeM4.fin.mistral-7b-v03.v32768` (vocab=32768, langs=[fin_Latn], iters=5,
em_mode=soft, byte_fallback=True, seed-fix applied).

## Loading

```python
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")
```