---
library_name: tokenizers
language:
- hun
tags:
- LangMAP
- unsupervised
- tokenizer
---

# Base + Language-Specific LangMAP — mistral-7b-v03 × hun_Latn

Unsupervised tokenization specialised for **hun_Latn**, derived from the
**mistral-7b-v03** base BPE tokenizer using the LangMAP framework.

This repository bundles:
- `base_tokenizer.json` — joint LAS Unigram base
- `langspec_hun_Latn.json` — language-specific overlay (re-EM on hun_Latn corpus)
- `tokenizer.json` — alias for the overlay (default load target)

Inference uses base + language-specific scores together (the LangMAP variant);
do not use the bare overlay or base on its own.

Trained from job `smoke.hun.mistral-7b-v03.v32768` (vocab=32768, langs=[hun_Latn], iters=5,
em_mode=soft, byte_fallback=True, seed-fix applied).

## Loading

```python
from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")
```