Upload Base + Language-Specific LangMAP mistral-7b-v03×fin_Latn (smoke smokeM4.fin.mistral-7b-v03.v32768)

README.md

@@ -0,0 +1,32 @@
+---
+library_name: tokenizers
+language:
+- fin
+tags:
+- LangMAP
+- unsupervised
+- tokenizer
+---
+
+# Base + Language-Specific LangMAP — mistral-7b-v03 × fin_Latn
+
+Unsupervised tokenization specialised for **fin_Latn**, derived from the
+**mistral-7b-v03** base BPE tokenizer using the LangMAP framework.
+
+This repository bundles:
+- `base_tokenizer.json` — joint LAS Unigram base
+- `langspec_fin_Latn.json` — language-specific overlay (re-EM on fin_Latn corpus)
+- `tokenizer.json` — alias for the overlay (default load target)
+
+Inference uses base + language-specific scores together (the LangMAP variant);
+do not use the bare overlay or base on its own.
+
+Trained from job `smokeM4.fin.mistral-7b-v03.v32768` (vocab=32768, langs=[fin_Latn], iters=5,
+em_mode=soft, byte_fallback=True, seed-fix applied).
+
+## Loading
+
+```python
+from tokenizers import Tokenizer
+tok = Tokenizer.from_file("tokenizer.json")
+```

metadata/base_tokenizer_prune_1.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8050599ea8cdb52e5ea6b0b847b5f1f34244a828367235f120a05bbf9f9ba707
+size 3328254