Upload Base + Language-Specific LangMAP gemma2-2b×fin_Latn (smoke smoke.fin.gemma2-2b.v256000)

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+base_tokenizer.json filter=lfs diff=lfs merge=lfs -text
+langspec_fin_Latn.json filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,32 @@
+---
+library_name: tokenizers
+language:
+- fin
+tags:
+- LangMAP
+- unsupervised
+- tokenizer
+---
+
+# Base + Language-Specific LangMAP — gemma2-2b × fin_Latn
+
+Unsupervised tokenization specialised for **fin_Latn**, derived from the
+**gemma2-2b** base BPE tokenizer using the LangMAP framework.
+
+This repository bundles:
+- `base_tokenizer.json` — joint LAS Unigram base
+- `langspec_fin_Latn.json` — language-specific overlay (re-EM on fin_Latn corpus)
+- `tokenizer.json` — alias for the overlay (default load target)
+
+Inference uses base + language-specific scores together (the LangMAP variant);
+do not use the bare overlay or base on its own.
+
+Trained from job `smoke.fin.gemma2-2b.v256000` (vocab=256000, langs=[fin_Latn], iters=5,
+em_mode=soft, byte_fallback=True, seed-fix applied).
+
+## Loading
+
+```python
+from tokenizers import Tokenizer
+tok = Tokenizer.from_file("tokenizer.json")
+```

base_tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dacfde4787b1e52e2c95f7f6769e796a2a4cb6f0d8ae13393693a2ba4063ebba
+size 16480687

langspec_fin_Latn.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a12f536c2fcb00f84067dbbb10c9f00972b2a636ec55b63fff0c95c453e3a051
+size 16356694

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a12f536c2fcb00f84067dbbb10c9f00972b2a636ec55b63fff0c95c453e3a051
+size 16356694