feat: Initial release of EthioBBPE - Ethiopian Language Tokenizer

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ታዲዮስ || Tadiyos Aschalew
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,3 +1,119 @@
----
-license: mit
----
+# 🇪🇹 EthioBBPE: Byte-Level BPE Tokenizer for Ethiopian Languages
+
+A robust, production-ready Byte-Level BPE (BBPE) tokenizer training framework specifically optimized for Ethiopian languages (Amharic, Oromo, Tigrinya, Somali, etc.) using Hugging Face's `tokenizers` library.
+
+## ✨ Features
+
+- **Optimized for Ethiopic Script**: Handles complex Ge'ez script characters and Latin-based orthographies seamlessly.
+- **Byte-Level Encoding**: Robust against unknown characters and emojis, ensuring no `<UNK>` tokens.
+- **End-to-End Pipeline**: From raw text corpus to a ready-to-use `tokenizer.json`.
+- **Hugging Face Compatible**: Directly usable with `transformers` models.
+- **Flexible Configuration**: Customize vocabulary size, minimum frequency, and special tokens.
+- **Multi-Format Support**: Train on `.txt`, `.json`, or `.jsonl` datasets.
+
+## 📦 Installation
+
+```bash
+pip install -r requirements.txt
+```
+
+## 🚀 Quick Start
+
+### 1. Prepare Your Data
+Place your training corpus (raw text files) in the `data/` directory.
+```text
+data/
+├── amharic_corpus.txt
+├── oromo_corpus.txt
+└── mixed_ethio_text.txt
+```
+
+### 2. Train the Tokenizer
+
+**Using CLI:**
+```bash
+python scripts/train_tokenizer.py \
+    --data_dir ./data \
+    --model_name EthioBBPE \
+    --vocab_size 32000 \
+    --min_frequency 2 \
+    --special_tokens "[PAD]","[UNK]","[CLS]","[SEP]","[MASK]"
+```
+
+**Using Python API:**
+```python
+from scripts.bbpe_trainer import BBPETrainer, BBPEConfig
+
+# Configure for Ethiopian Languages
+config = BBPEConfig(
+    vocab_size=32000,
+    min_frequency=2,
+    show_progress=True,
+    special_tokens=["[PAD]", "[UNK]", "[CLS]", "[SEP]", "[MASK]"]
+)
+
+trainer = BBPETrainer(config=config, model_name="EthioBBPE")
+trainer.train_from_directory("./data")
+trainer.save("./models/EthioBBPE")
+
+# Test it
+text = "ሰላም ልጄ እንዴት ነሽ? (Hello daughter, how are you?)"
+tokens = trainer.tokenize(text)
+print(f"Tokens: {tokens}")
+```
+
+### 3. Load and Use
+
+```python
+from tokenizers import Tokenizer
+
+# Load the trained tokenizer
+tokenizer = Tokenizer.from_file("models/EthioBBPE/tokenizer.json")
+
+# Encode
+encoded = tokenizer.encode("የኢትዮጵያ ህዝብ")
+print(encoded.ids)
+print(encoded.tokens)
+
+# Decode
+decoded = tokenizer.decode(encoded.ids)
+print(decoded)
+```
+
+## 🏗️ Architecture
+
+The `EthioBBPE` architecture follows these steps:
+1. **Pre-tokenization**: Splits text into words while preserving byte-level integrity.
+2. **Byte Conversion**: Converts all characters (including Ge'ez script) into their byte representations.
+3. **BPE Training**: Learns merge operations based on frequency in the corpus.
+4. **Vocabulary Creation**: Generates a fixed-size vocabulary of byte-level tokens.
+
+## 📂 Project Structure
+
+```text
+Ethio_BBPE/
+├── data/                   # Raw training data
+├── models/                 # Output directory for trained models
+├── scripts/
+│   ├── bbpe_trainer.py     # Core logic (BBPEConfig, BBPETrainer)
+│   ├── train_tokenizer.py  # CLI entry point
+│   └── example_usage.py    # Usage examples
+├── requirements.txt        # Dependencies
+└── README.md               # This file
+```
+
+## 🤗 Hugging Face Hub
+
+This model is designed to be pushed to the Hugging Face Hub:
+
+```python
+trainer.push_to_hub("Nexuss0781/Ethio-BBPE", token="YOUR_HF_TOKEN")
+```
+
+## 📄 License
+
+MIT License
+
+## 🙏 Acknowledgments
+
+Built for the Ethiopian NLP community to foster better language understanding and generation capabilities.

data/sample_corpus.txt

@@ -0,0 +1,33 @@
+This is a sample training corpus for the BBPE tokenizer.
+
+The quick brown fox jumps over the lazy dog. This sentence contains every letter of the alphabet.
+
+Natural Language Processing (NLP) is a field of artificial intelligence that focuses on the interaction between computers and humans through language. The ultimate objective of NLP is to read, decipher, understand, and make sense of human languages in a manner that is valuable.
+
+Machine learning is a subset of artificial intelligence that provides systems the ability to learn and improve from experience without being explicitly programmed. Machine learning focuses on the development of computer programs that can access data and use it to learn for themselves.
+
+Deep learning is part of a broader family of machine learning methods based on artificial neural networks with representation learning. Learning can be supervised, semi-supervised or unsupervised.
+
+Tokenization is the process of splitting text into smaller units called tokens. These tokens can be words, characters, or subwords. Byte-level BPE (Byte Pair Encoding) is a popular tokenization algorithm used in modern language models.
+
+The transformer architecture has revolutionized natural language processing. It uses self-attention mechanisms to process input sequences in parallel, making it much faster than recurrent neural networks.
+
+BERT, GPT, and T5 are some of the most famous transformer-based models. They have achieved state-of-the-art results on various NLP tasks including question answering, text classification, and machine translation.
+
+Training a tokenizer requires a representative corpus of text. The quality and diversity of your training data directly impacts the performance of your tokenizer.
+
+Special tokens like [PAD], [UNK], [CLS], and [SEP] are often added to vocabularies for specific tasks. These tokens help models handle padding, unknown words, and sequence boundaries.
+
+The vocabulary size is an important hyperparameter. Too small and you'll have many unknown tokens. Too large and your model becomes inefficient. Typical sizes range from 20,000 to 100,000 tokens.
+
+Byte-level encoding ensures that any Unicode text can be tokenized without errors. This is particularly important for multilingual applications.
+
+Minimum frequency threshold helps filter out rare tokens that might be noise. A typical value is 2 or 3, meaning a token must appear at least that many times to be included in the vocabulary.
+
+Prefix space is often added before tokenization to distinguish between words at the beginning of sentences and words in other positions.
+
+This sample text provides enough content to demonstrate the BBPE tokenizer training process. In practice, you would use much larger corpora containing millions or billions of tokens.
+
+Remember to always validate your tokenizer on held-out data to ensure it generalizes well to unseen text.
+
+Happy tokenizing!

models/demo_tokenizer/config.json

@@ -0,0 +1,19 @@
+{
+  "vocab_size": 30000,
+  "min_frequency": 2,
+  "special_tokens": [
+    "<pad>",
+    "<unk>",
+    "<s>",
+    "</s>",
+    "<mask>"
+  ],
+  "lowercase": false,
+  "add_prefix_space": false,
+  "trim_offsets": true,
+  "show_progress": true,
+  "initial_alphabet": [],
+  "data_dir": "data",
+  "model_save_dir": "models",
+  "model_name": "demo_tokenizer"
+}

models/demo_tokenizer/merges.txt

@@ -0,0 +1,329 @@
+#version: 0.2
+Ġ t
+i n
+e n
+Ġ a
+e r
+e s
+Ġ o
+Ġt o
+a n
+in g
+a r
+a t
+h e
+i s
+o r
+Ġ m
+Ġ s
+a l
+k en
+o n
+Ġto ken
+l e
+Ġo f
+Ġ b
+Ġ p
+i t
+o u
+Ġ c
+Ġ f
+Ġa n
+i c
+i z
+i on
+r o
+e d
+e l
+r e
+Ġ in
+Ġt he
+Ġ is
+Ġan d
+Ġ T
+Ġ u
+Ġb e
+Ġ d
+Ġt h
+a c
+m p
+o d
+Ġ w
+Ġ le
+en c
+en t
+Ġtoken iz
+a s
+e x
+t er
+Ġ l
+ar n
+Ġtoken s
+Ġp ro
+q u
+u l
+Ġ h
+Ġ v
+at ion
+Ġth at
+Ġle arn
+a in
+c es
+e c
+f ic
+g e
+i l
+r es
+u a
+y ou
+Ġ n
+Ġ re
+Ġ you
+Ġt r
+Ġt ex
+Ġu n
+ces s
+Ġtex t
+0 0
+a m
+d s
+e m
+e t
+g ua
+i fic
+o c
+p er
+s t
+u r
+v e
+Ġo n
+an gua
+or ds
+Ġm od
+enc e
+Ġtokeniz er
+ul ar
+Ġlearn ing
+ain ing
+a b
+c h
+f or
+h in
+h is
+i al
+i mp
+k s
+s e
+s u
+u s
+Ġ B
+Ġ [
+Ġ imp
+Ġa r
+Ġo r
+an s
+at e
+it h
+Ġc an
+Ġf or
+ac hin
+Ġw ords
+Ġl angua
+Ġpro cess
+Ġyou r
+ur al
+Ġmod el
+achin e
+L P
+N LP
+P E
+T he
+] ,
+a d
+d ed
+e v
+g h
+i d
+i m
+l d
+l p
+l u
+l y
+p l
+t e
+t ion
+t ific
+u m
+v is
+w n
+y te
+Ġ en
+Ġ he
+Ġ it
+Ġ qu
+ar t
+at a
+or p
+Ġm u
+Ġm an
+Ġs u
+on t
+it y
+ou t
+Ġc orp
+Ġc ont
+ion s
+el l
+Ġin t
+ĠT he
+Ġd ata
+Ġw ith
+Ġv al
+Ġv oc
+Ġtr aining
+Ġtr ans
+per vis
+ab ular
+for m
+Ġar tific
+Ġlangua ge
+Ġmodel s
+Ġvoc abular
+pervis ed
+Ġartific ial
+B PE
+B yte
+M achine
+T his
+a k
+a mp
+a ve
+c lu
+d ing
+d ded
+e en
+e qu
+e ural
+g r
+g ence
+h er
+h es
+i es
+i ve
+i gence
+k n
+l ions
+m s
+m al
+o m
+o o
+o p
+o mp
+o wn
+p p
+p ic
+p ec
+p res
+t s
+t w
+t en
+t an
+t ing
+u t
+u ter
+v er
+v el
+v id
+w or
+y pic
+Ġ (
+Ġ 2
+Ġ I
+Ġ L
+Ġ P
+Ġ r
+Ġ ex
+Ġ NLP
+Ġt as
+Ġa l
+Ġa t
+Ġa re
+Ġa dded
+an d
+at ural
+or tan
+Ġm achine
+Ġm ak
+Ġs iz
+Ġs ent
+Ġs amp
+Ġs equ
+Ġs mal
+al le
+le vel
+Ġof ten
+Ġb ro
+Ġp art
+ou s
+ou gh
+Ġc omp
+Ġf ro
+Ġf am
+Ġf oc
+Ġin clu
+ĠT his
+ĠT hes
+ĠT oo
+Ġu s
+Ġu se
+Ġbe tw
+Ġd i
+od ing
+enc es
+Ġtokeniz ation
+as ed
+Ġl ar
+Ġpro gr
+Ġpro vid
+Ġh um
+Ġh ave
+ec t
+il lions
+Ġn et
+Ġn eural
+Ġre pres
+Ġun kn
+00 0
+su pervised
+us es
+ĠB BPE
+Ġimp ortan
+pl ic
+Ġen su
+Ġhe lp
+Ġmu ch
+Ġman y
+Ġsu b
+Ġcorp us
+ell igence
+Ġint elligence
+Ġwith out
+Ġtrans form
+Ġvocabular y
+wor ks
+ypic al
+Ġtas ks
+Ġsamp le
+Ġsmal l
+Ġcomp uter
+Ġfro m
+Ġfoc uses
+ĠThes e
+Ġbetw een
+Ġprogr am
+Ġprovid es
+Ġnet works
+Ġrepres ent
+Ġunkn own
+Ġimportan t
+Ġtransform er

models/demo_tokenizer/tokenizer.json

@@ -0,0 +1,1986 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<pad>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<unk>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "<s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "</s>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 4,
+      "content": "<mask>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "ByteLevel",
+    "add_prefix_space": false,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "post_processor": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "decoder": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "model": {
+    "type": "BPE",
+    "dropout": null,
+    "unk_token": null,
+    "continuing_subword_prefix": null,
+    "end_of_word_suffix": null,
+    "fuse_unk": false,
+    "byte_fallback": false,
+    "ignore_merges": false,
+    "vocab": {
+      "<pad>": 0,
+      "<unk>": 1,
+      "<s>": 2,
+      "</s>": 3,
+      "<mask>": 4,
+      "!": 5,
+      "\"": 6,
+      "#": 7,
+      "$": 8,
+      "%": 9,
+      "&": 10,
+      "'": 11,
+      "(": 12,
+      ")": 13,
+      "*": 14,
+      "+": 15,
+      ",": 16,
+      "-": 17,
+      ".": 18,
+      "/": 19,
+      "0": 20,
+      "1": 21,
+      "2": 22,
+      "3": 23,
+      "4": 24,
+      "5": 25,
+      "6": 26,
+      "7": 27,
+      "8": 28,
+      "9": 29,
+      ":": 30,
+      ";": 31,
+      "<": 32,
+      "=": 33,
+      ">": 34,
+      "?": 35,
+      "@": 36,
+      "A": 37,
+      "B": 38,
+      "C": 39,
+      "D": 40,
+      "E": 41,
+      "F": 42,
+      "G": 43,
+      "H": 44,
+      "I": 45,
+      "J": 46,
+      "K": 47,
+      "L": 48,
+      "M": 49,
+      "N": 50,
+      "O": 51,
+      "P": 52,
+      "Q": 53,
+      "R": 54,
+      "S": 55,
+      "T": 56,
+      "U": 57,
+      "V": 58,
+      "W": 59,
+      "X": 60,
+      "Y": 61,
+      "Z": 62,
+      "[": 63,
+      "\\": 64,
+      "]": 65,
+      "^": 66,
+      "_": 67,
+      "`": 68,
+      "a": 69,
+      "b": 70,
+      "c": 71,
+      "d": 72,
+      "e": 73,
+      "f": 74,
+      "g": 75,
+      "h": 76,
+      "i": 77,
+      "j": 78,
+      "k": 79,
+      "l": 80,
+      "m": 81,
+      "n": 82,
+      "o": 83,
+      "p": 84,
+      "q": 85,
+      "r": 86,
+      "s": 87,
+      "t": 88,
+      "u": 89,
+      "v": 90,
+      "w": 91,
+      "x": 92,
+      "y": 93,
+      "z": 94,
+      "{": 95,
+      "|": 96,
+      "}": 97,
+      "~": 98,
+      "¡": 99,
+      "¢": 100,
+      "£": 101,
+      "¤": 102,
+      "¥": 103,
+      "¦": 104,
+      "§": 105,
+      "¨": 106,
+      "©": 107,
+      "ª": 108,
+      "«": 109,
+      "¬": 110,
+      "®": 111,
+      "¯": 112,
+      "°": 113,
+      "±": 114,
+      "²": 115,
+      "³": 116,
+      "´": 117,
+      "µ": 118,
+      "¶": 119,
+      "·": 120,
+      "¸": 121,
+      "¹": 122,
+      "º": 123,
+      "»": 124,
+      "¼": 125,
+      "½": 126,
+      "¾": 127,
+      "¿": 128,
+      "À": 129,
+      "Á": 130,
+      "Â": 131,
+      "Ã": 132,
+      "Ä": 133,
+      "Å": 134,
+      "Æ": 135,
+      "Ç": 136,
+      "È": 137,
+      "É": 138,
+      "Ê": 139,
+      "Ë": 140,
+      "Ì": 141,
+      "Í": 142,
+      "Î": 143,
+      "Ï": 144,
+      "Ð": 145,
+      "Ñ": 146,
+      "Ò": 147,
+      "Ó": 148,
+      "Ô": 149,
+      "Õ": 150,
+      "Ö": 151,
+      "×": 152,
+      "Ø": 153,
+      "Ù": 154,
+      "Ú": 155,
+      "Û": 156,
+      "Ü": 157,
+      "Ý": 158,
+      "Þ": 159,
+      "ß": 160,
+      "à": 161,
+      "á": 162,
+      "â": 163,
+      "ã": 164,
+      "ä": 165,
+      "å": 166,
+      "æ": 167,
+      "ç": 168,
+      "è": 169,
+      "é": 170,
+      "ê": 171,
+      "ë": 172,
+      "ì": 173,
+      "í": 174,
+      "î": 175,
+      "ï": 176,
+      "ð": 177,
+      "ñ": 178,
+      "ò": 179,
+      "ó": 180,
+      "ô": 181,
+      "õ": 182,
+      "ö": 183,
+      "÷": 184,
+      "ø": 185,
+      "ù": 186,
+      "ú": 187,
+      "û": 188,
+      "ü": 189,
+      "ý": 190,
+      "þ": 191,
+      "ÿ": 192,
+      "Ā": 193,
+      "ā": 194,
+      "Ă": 195,
+      "ă": 196,
+      "Ą": 197,
+      "ą": 198,
+      "Ć": 199,
+      "ć": 200,
+      "Ĉ": 201,
+      "ĉ": 202,
+      "Ċ": 203,
+      "ċ": 204,
+      "Č": 205,
+      "č": 206,
+      "Ď": 207,
+      "ď": 208,
+      "Đ": 209,
+      "đ": 210,
+      "Ē": 211,
+      "ē": 212,
+      "Ĕ": 213,
+      "ĕ": 214,
+      "Ė": 215,
+      "ė": 216,
+      "Ę": 217,
+      "ę": 218,
+      "Ě": 219,
+      "ě": 220,
+      "Ĝ": 221,
+      "ĝ": 222,
+      "Ğ": 223,
+      "ğ": 224,
+      "Ġ": 225,
+      "ġ": 226,
+      "Ģ": 227,
+      "ģ": 228,
+      "Ĥ": 229,
+      "ĥ": 230,
+      "Ħ": 231,
+      "ħ": 232,
+      "Ĩ": 233,
+      "ĩ": 234,
+      "Ī": 235,
+      "ī": 236,
+      "Ĭ": 237,
+      "ĭ": 238,
+      "Į": 239,
+      "į": 240,
+      "İ": 241,
+      "ı": 242,
+      "IJ": 243,
+      "ij": 244,
+      "Ĵ": 245,
+      "ĵ": 246,
+      "Ķ": 247,
+      "ķ": 248,
+      "ĸ": 249,
+      "Ĺ": 250,
+      "ĺ": 251,
+      "Ļ": 252,
+      "ļ": 253,
+      "Ľ": 254,
+      "ľ": 255,
+      "Ŀ": 256,
+      "ŀ": 257,
+      "Ł": 258,
+      "ł": 259,
+      "Ń": 260,
+      "Ġt": 261,
+      "in": 262,
+      "en": 263,
+      "Ġa": 264,
+      "er": 265,
+      "es": 266,
+      "Ġo": 267,
+      "Ġto": 268,
+      "an": 269,
+      "ing": 270,
+      "ar": 271,
+      "at": 272,
+      "he": 273,
+      "is": 274,
+      "or": 275,
+      "Ġm": 276,
+      "Ġs": 277,
+      "al": 278,
+      "ken": 279,
+      "on": 280,
+      "Ġtoken": 281,
+      "le": 282,
+      "Ġof": 283,
+      "Ġb": 284,
+      "Ġp": 285,
+      "it": 286,
+      "ou": 287,
+      "Ġc": 288,
+      "Ġf": 289,
+      "Ġan": 290,
+      "ic": 291,
+      "iz": 292,
+      "ion": 293,
+      "ro": 294,
+      "ed": 295,
+      "el": 296,
+      "re": 297,
+      "Ġin": 298,
+      "Ġthe": 299,
+      "Ġis": 300,
+      "Ġand": 301,
+      "ĠT": 302,
+      "Ġu": 303,
+      "Ġbe": 304,
+      "Ġd": 305,
+      "Ġth": 306,
+      "ac": 307,
+      "mp": 308,
+      "od": 309,
+      "Ġw": 310,
+      "Ġle": 311,
+      "enc": 312,
+      "ent": 313,
+      "Ġtokeniz": 314,
+      "as": 315,
+      "ex": 316,
+      "ter": 317,
+      "Ġl": 318,
+      "arn": 319,
+      "Ġtokens": 320,
+      "Ġpro": 321,
+      "qu": 322,
+      "ul": 323,
+      "Ġh": 324,
+      "Ġv": 325,
+      "ation": 326,
+      "Ġthat": 327,
+      "Ġlearn": 328,
+      "ain": 329,
+      "ces": 330,
+      "ec": 331,
+      "fic": 332,
+      "ge": 333,
+      "il": 334,
+      "res": 335,
+      "ua": 336,
+      "you": 337,
+      "Ġn": 338,
+      "Ġre": 339,
+      "Ġyou": 340,
+      "Ġtr": 341,
+      "Ġtex": 342,
+      "Ġun": 343,
+      "cess": 344,
+      "Ġtext": 345,
+      "00": 346,
+      "am": 347,
+      "ds": 348,
+      "em": 349,
+      "et": 350,
+      "gua": 351,
+      "ific": 352,
+      "oc": 353,
+      "per": 354,
+      "st": 355,
+      "ur": 356,
+      "ve": 357,
+      "Ġon": 358,
+      "angua": 359,
+      "ords": 360,
+      "Ġmod": 361,
+      "ence": 362,
+      "Ġtokenizer": 363,
+      "ular": 364,
+      "Ġlearning": 365,
+      "aining": 366,
+      "ab": 367,
+      "ch": 368,
+      "for": 369,
+      "hin": 370,
+      "his": 371,
+      "ial": 372,
+      "imp": 373,
+      "ks": 374,
+      "se": 375,
+      "su": 376,
+      "us": 377,
+      "ĠB": 378,
+      "Ġ[": 379,
+      "Ġimp": 380,
+      "Ġar": 381,
+      "Ġor": 382,
+      "ans": 383,
+      "ate": 384,
+      "ith": 385,
+      "Ġcan": 386,
+      "Ġfor": 387,
+      "achin": 388,
+      "Ġwords": 389,
+      "Ġlangua": 390,
+      "Ġprocess": 391,
+      "Ġyour": 392,
+      "ural": 393,
+      "Ġmodel": 394,
+      "achine": 395,
+      "LP": 396,
+      "NLP": 397,
+      "PE": 398,
+      "The": 399,
+      "],": 400,
+      "ad": 401,
+      "ded": 402,
+      "ev": 403,
+      "gh": 404,
+      "id": 405,
+      "im": 406,
+      "ld": 407,
+      "lp": 408,
+      "lu": 409,
+      "ly": 410,
+      "pl": 411,
+      "te": 412,
+      "tion": 413,
+      "tific": 414,
+      "um": 415,
+      "vis": 416,
+      "wn": 417,
+      "yte": 418,
+      "Ġen": 419,
+      "Ġhe": 420,
+      "Ġit": 421,
+      "Ġqu": 422,
+      "art": 423,
+      "ata": 424,
+      "orp": 425,
+      "Ġmu": 426,
+      "Ġman": 427,
+      "Ġsu": 428,
+      "ont": 429,
+      "ity": 430,
+      "out": 431,
+      "Ġcorp": 432,
+      "Ġcont": 433,
+      "ions": 434,
+      "ell": 435,
+      "Ġint": 436,
+      "ĠThe": 437,
+      "Ġdata": 438,
+      "Ġwith": 439,
+      "Ġval": 440,
+      "Ġvoc": 441,
+      "Ġtraining": 442,
+      "Ġtrans": 443,
+      "pervis": 444,
+      "abular": 445,
+      "form": 446,
+      "Ġartific": 447,
+      "Ġlanguage": 448,
+      "Ġmodels": 449,
+      "Ġvocabular": 450,
+      "pervised": 451,
+      "Ġartificial": 452,
+      "BPE": 453,
+      "Byte": 454,
+      "Machine": 455,
+      "This": 456,
+      "ak": 457,
+      "amp": 458,
+      "ave": 459,
+      "clu": 460,
+      "ding": 461,
+      "dded": 462,
+      "een": 463,
+      "equ": 464,
+      "eural": 465,
+      "gr": 466,
+      "gence": 467,
+      "her": 468,
+      "hes": 469,
+      "ies": 470,
+      "ive": 471,
+      "igence": 472,
+      "kn": 473,
+      "lions": 474,
+      "ms": 475,
+      "mal": 476,
+      "om": 477,
+      "oo": 478,
+      "op": 479,
+      "omp": 480,
+      "own": 481,
+      "pp": 482,
+      "pic": 483,
+      "pec": 484,
+      "pres": 485,
+      "ts": 486,
+      "tw": 487,
+      "ten": 488,
+      "tan": 489,
+      "ting": 490,
+      "ut": 491,
+      "uter": 492,
+      "ver": 493,
+      "vel": 494,
+      "vid": 495,
+      "wor": 496,
+      "ypic": 497,
+      "Ġ(": 498,
+      "Ġ2": 499,
+      "ĠI": 500,
+      "ĠL": 501,
+      "ĠP": 502,
+      "Ġr": 503,
+      "Ġex": 504,
+      "ĠNLP": 505,
+      "Ġtas": 506,
+      "Ġal": 507,
+      "Ġat": 508,
+      "Ġare": 509,
+      "Ġadded": 510,
+      "and": 511,
+      "atural": 512,
+      "ortan": 513,
+      "Ġmachine": 514,
+      "Ġmak": 515,
+      "Ġsiz": 516,
+      "Ġsent": 517,
+      "Ġsamp": 518,
+      "Ġsequ": 519,
+      "Ġsmal": 520,
+      "alle": 521,
+      "level": 522,
+      "Ġoften": 523,
+      "Ġbro": 524,
+      "Ġpart": 525,
+      "ous": 526,
+      "ough": 527,
+      "Ġcomp": 528,
+      "Ġfro": 529,
+      "Ġfam": 530,
+      "Ġfoc": 531,
+      "Ġinclu": 532,
+      "ĠThis": 533,
+      "ĠThes": 534,
+      "ĠToo": 535,
+      "Ġus": 536,
+      "Ġuse": 537,
+      "Ġbetw": 538,
+      "Ġdi": 539,
+      "oding": 540,
+      "ences": 541,
+      "Ġtokenization": 542,
+      "ased": 543,
+      "Ġlar": 544,
+      "Ġprogr": 545,
+      "Ġprovid": 546,
+      "Ġhum": 547,
+      "Ġhave": 548,
+      "ect": 549,
+      "illions": 550,
+      "Ġnet": 551,
+      "Ġneural": 552,
+      "Ġrepres": 553,
+      "Ġunkn": 554,
+      "000": 555,
+      "supervised": 556,
+      "uses": 557,
+      "ĠBBPE": 558,
+      "Ġimportan": 559,
+      "plic": 560,
+      "Ġensu": 561,
+      "Ġhelp": 562,
+      "Ġmuch": 563,
+      "Ġmany": 564,
+      "Ġsub": 565,
+      "Ġcorpus": 566,
+      "elligence": 567,
+      "Ġintelligence": 568,
+      "Ġwithout": 569,
+      "Ġtransform": 570,
+      "Ġvocabulary": 571,
+      "works": 572,
+      "ypical": 573,
+      "Ġtasks": 574,
+      "Ġsample": 575,
+      "Ġsmall": 576,
+      "Ġcomputer": 577,
+      "Ġfrom": 578,
+      "Ġfocuses": 579,
+      "ĠThese": 580,
+      "Ġbetween": 581,
+      "Ġprogram": 582,
+      "Ġprovides": 583,
+      "Ġnetworks": 584,
+      "Ġrepresent": 585,
+      "Ġunknown": 586,
+      "Ġimportant": 587,
+      "Ġtransformer": 588
+    },
+    "merges": [
+      [
+        "Ġ",
+        "t"
+      ],
+      [
+        "i",
+        "n"
+      ],
+      [
+        "e",
+        "n"
+      ],
+      [
+        "Ġ",
+        "a"
+      ],
+      [
+        "e",
+        "r"
+      ],
+      [
+        "e",
+        "s"
+      ],
+      [
+        "Ġ",
+        "o"
+      ],
+      [
+        "Ġt",
+        "o"
+      ],
+      [
+        "a",
+        "n"
+      ],
+      [
+        "in",
+        "g"
+      ],
+      [
+        "a",
+        "r"
+      ],
+      [
+        "a",
+        "t"
+      ],
+      [
+        "h",
+        "e"
+      ],
+      [
+        "i",
+        "s"
+      ],
+      [
+        "o",
+        "r"
+      ],
+      [
+        "Ġ",
+        "m"
+      ],
+      [
+        "Ġ",
+        "s"
+      ],
+      [
+        "a",
+        "l"
+      ],
+      [
+        "k",
+        "en"
+      ],
+      [
+        "o",
+        "n"
+      ],
+      [
+        "Ġto",
+        "ken"
+      ],
+      [
+        "l",
+        "e"
+      ],
+      [
+        "Ġo",
+        "f"
+      ],
+      [
+        "Ġ",
+        "b"
+      ],
+      [
+        "Ġ",
+        "p"
+      ],
+      [
+        "i",
+        "t"
+      ],
+      [
+        "o",
+        "u"
+      ],
+      [
+        "Ġ",
+        "c"
+      ],
+      [
+        "Ġ",
+        "f"
+      ],
+      [
+        "Ġa",
+        "n"
+      ],
+      [
+        "i",
+        "c"
+      ],
+      [
+        "i",
+        "z"
+      ],
+      [
+        "i",
+        "on"
+      ],
+      [
+        "r",
+        "o"
+      ],
+      [
+        "e",
+        "d"
+      ],
+      [
+        "e",
+        "l"
+      ],
+      [
+        "r",
+        "e"
+      ],
+      [
+        "Ġ",
+        "in"
+      ],
+      [
+        "Ġt",
+        "he"
+      ],
+      [
+        "Ġ",
+        "is"
+      ],
+      [
+        "Ġan",
+        "d"
+      ],
+      [
+        "Ġ",
+        "T"
+      ],
+      [
+        "Ġ",
+        "u"
+      ],
+      [
+        "Ġb",
+        "e"
+      ],
+      [
+        "Ġ",
+        "d"
+      ],
+      [
+        "Ġt",
+        "h"
+      ],
+      [
+        "a",
+        "c"
+      ],
+      [
+        "m",
+        "p"
+      ],
+      [
+        "o",
+        "d"
+      ],
+      [
+        "Ġ",
+        "w"
+      ],
+      [
+        "Ġ",
+        "le"
+      ],
+      [
+        "en",
+        "c"
+      ],
+      [
+        "en",
+        "t"
+      ],
+      [
+        "Ġtoken",
+        "iz"
+      ],
+      [
+        "a",
+        "s"
+      ],
+      [
+        "e",
+        "x"
+      ],
+      [
+        "t",
+        "er"
+      ],
+      [
+        "Ġ",
+        "l"
+      ],
+      [
+        "ar",
+        "n"
+      ],
+      [
+        "Ġtoken",
+        "s"
+      ],
+      [
+        "Ġp",
+        "ro"
+      ],
+      [
+        "q",
+        "u"
+      ],
+      [
+        "u",
+        "l"
+      ],
+      [
+        "Ġ",
+        "h"
+      ],
+      [
+        "Ġ",
+        "v"
+      ],
+      [
+        "at",
+        "ion"
+      ],
+      [
+        "Ġth",
+        "at"
+      ],
+      [
+        "Ġle",
+        "arn"
+      ],
+      [
+        "a",
+        "in"
+      ],
+      [
+        "c",
+        "es"
+      ],
+      [
+        "e",
+        "c"
+      ],
+      [
+        "f",
+        "ic"
+      ],
+      [
+        "g",
+        "e"
+      ],
+      [
+        "i",
+        "l"
+      ],
+      [
+        "r",
+        "es"
+      ],
+      [
+        "u",
+        "a"
+      ],
+      [
+        "y",
+        "ou"
+      ],
+      [
+        "Ġ",
+        "n"
+      ],
+      [
+        "Ġ",
+        "re"
+      ],
+      [
+        "Ġ",
+        "you"
+      ],
+      [
+        "Ġt",
+        "r"
+      ],
+      [
+        "Ġt",
+        "ex"
+      ],
+      [
+        "Ġu",
+        "n"
+      ],
+      [
+        "ces",
+        "s"
+      ],
+      [
+        "Ġtex",
+        "t"
+      ],
+      [
+        "0",
+        "0"
+      ],
+      [
+        "a",
+        "m"
+      ],
+      [
+        "d",
+        "s"
+      ],
+      [
+        "e",
+        "m"
+      ],
+      [
+        "e",
+        "t"
+      ],
+      [
+        "g",
+        "ua"
+      ],
+      [
+        "i",
+        "fic"
+      ],
+      [
+        "o",
+        "c"
+      ],
+      [
+        "p",
+        "er"
+      ],
+      [
+        "s",
+        "t"
+      ],
+      [
+        "u",
+        "r"
+      ],
+      [
+        "v",
+        "e"
+      ],
+      [
+        "Ġo",
+        "n"
+      ],
+      [
+        "an",
+        "gua"
+      ],
+      [
+        "or",
+        "ds"
+      ],
+      [
+        "Ġm",
+        "od"
+      ],
+      [
+        "enc",
+        "e"
+      ],
+      [
+        "Ġtokeniz",
+        "er"
+      ],
+      [
+        "ul",
+        "ar"
+      ],
+      [
+        "Ġlearn",
+        "ing"
+      ],
+      [
+        "ain",
+        "ing"
+      ],
+      [
+        "a",
+        "b"
+      ],
+      [
+        "c",
+        "h"
+      ],
+      [
+        "f",
+        "or"
+      ],
+      [
+        "h",
+        "in"
+      ],
+      [
+        "h",
+        "is"
+      ],
+      [
+        "i",
+        "al"
+      ],
+      [
+        "i",
+        "mp"
+      ],
+      [
+        "k",
+        "s"
+      ],
+      [
+        "s",
+        "e"
+      ],
+      [
+        "s",
+        "u"
+      ],
+      [
+        "u",
+        "s"
+      ],
+      [
+        "Ġ",
+        "B"
+      ],
+      [
+        "Ġ",
+        "["
+      ],
+      [
+        "Ġ",
+        "imp"
+      ],
+      [
+        "Ġa",
+        "r"
+      ],
+      [
+        "Ġo",
+        "r"
+      ],
+      [
+        "an",
+        "s"
+      ],
+      [
+        "at",
+        "e"
+      ],
+      [
+        "it",
+        "h"
+      ],
+      [
+        "Ġc",
+        "an"
+      ],
+      [
+        "Ġf",
+        "or"
+      ],
+      [
+        "ac",
+        "hin"
+      ],
+      [
+        "Ġw",
+        "ords"
+      ],
+      [
+        "Ġl",
+        "angua"
+      ],
+      [
+        "Ġpro",
+        "cess"
+      ],
+      [
+        "Ġyou",
+        "r"
+      ],
+      [
+        "ur",
+        "al"
+      ],
+      [
+        "Ġmod",
+        "el"
+      ],
+      [
+        "achin",
+        "e"
+      ],
+      [
+        "L",
+        "P"
+      ],
+      [
+        "N",
+        "LP"
+      ],
+      [
+        "P",
+        "E"
+      ],
+      [
+        "T",
+        "he"
+      ],
+      [
+        "]",
+        ","
+      ],
+      [
+        "a",
+        "d"
+      ],
+      [
+        "d",
+        "ed"
+      ],
+      [
+        "e",
+        "v"
+      ],
+      [
+        "g",
+        "h"
+      ],
+      [
+        "i",
+        "d"
+      ],
+      [
+        "i",
+        "m"
+      ],
+      [
+        "l",
+        "d"
+      ],
+      [
+        "l",
+        "p"
+      ],
+      [
+        "l",
+        "u"
+      ],
+      [
+        "l",
+        "y"
+      ],
+      [
+        "p",
+        "l"
+      ],
+      [
+        "t",
+        "e"
+      ],
+      [
+        "t",
+        "ion"
+      ],
+      [
+        "t",
+        "ific"
+      ],
+      [
+        "u",
+        "m"
+      ],
+      [
+        "v",
+        "is"
+      ],
+      [
+        "w",
+        "n"
+      ],
+      [
+        "y",
+        "te"
+      ],
+      [
+        "Ġ",
+        "en"
+      ],
+      [
+        "Ġ",
+        "he"
+      ],
+      [
+        "Ġ",
+        "it"
+      ],
+      [
+        "Ġ",
+        "qu"
+      ],
+      [
+        "ar",
+        "t"
+      ],
+      [
+        "at",
+        "a"
+      ],
+      [
+        "or",
+        "p"
+      ],
+      [
+        "Ġm",
+        "u"
+      ],
+      [
+        "Ġm",
+        "an"
+      ],
+      [
+        "Ġs",
+        "u"
+      ],
+      [
+        "on",
+        "t"
+      ],
+      [
+        "it",
+        "y"
+      ],
+      [
+        "ou",
+        "t"
+      ],
+      [
+        "Ġc",
+        "orp"
+      ],
+      [
+        "Ġc",
+        "ont"
+      ],
+      [
+        "ion",
+        "s"
+      ],
+      [
+        "el",
+        "l"
+      ],
+      [
+        "Ġin",
+        "t"
+      ],
+      [
+        "ĠT",
+        "he"
+      ],
+      [
+        "Ġd",
+        "ata"
+      ],
+      [
+        "Ġw",
+        "ith"
+      ],
+      [
+        "Ġv",
+        "al"
+      ],
+      [
+        "Ġv",
+        "oc"
+      ],
+      [
+        "Ġtr",
+        "aining"
+      ],
+      [
+        "Ġtr",
+        "ans"
+      ],
+      [
+        "per",
+        "vis"
+      ],
+      [
+        "ab",
+        "ular"
+      ],
+      [
+        "for",
+        "m"
+      ],
+      [
+        "Ġar",
+        "tific"
+      ],
+      [
+        "Ġlangua",
+        "ge"
+      ],
+      [
+        "Ġmodel",
+        "s"
+      ],
+      [
+        "Ġvoc",
+        "abular"
+      ],
+      [
+        "pervis",
+        "ed"
+      ],
+      [
+        "Ġartific",
+        "ial"
+      ],
+      [
+        "B",
+        "PE"
+      ],
+      [
+        "B",
+        "yte"
+      ],
+      [
+        "M",
+        "achine"
+      ],
+      [
+        "T",
+        "his"
+      ],
+      [
+        "a",
+        "k"
+      ],
+      [
+        "a",
+        "mp"
+      ],
+      [
+        "a",
+        "ve"
+      ],
+      [
+        "c",
+        "lu"
+      ],
+      [
+        "d",
+        "ing"
+      ],
+      [
+        "d",
+        "ded"
+      ],
+      [
+        "e",
+        "en"
+      ],
+      [
+        "e",
+        "qu"
+      ],
+      [
+        "e",
+        "ural"
+      ],
+      [
+        "g",
+        "r"
+      ],
+      [
+        "g",
+        "ence"
+      ],
+      [
+        "h",
+        "er"
+      ],
+      [
+        "h",
+        "es"
+      ],
+      [
+        "i",
+        "es"
+      ],
+      [
+        "i",
+        "ve"
+      ],
+      [
+        "i",
+        "gence"
+      ],
+      [
+        "k",
+        "n"
+      ],
+      [
+        "l",
+        "ions"
+      ],
+      [
+        "m",
+        "s"
+      ],
+      [
+        "m",
+        "al"
+      ],
+      [
+        "o",
+        "m"
+      ],
+      [
+        "o",
+        "o"
+      ],
+      [
+        "o",
+        "p"
+      ],
+      [
+        "o",
+        "mp"
+      ],
+      [
+        "o",
+        "wn"
+      ],
+      [
+        "p",
+        "p"
+      ],
+      [
+        "p",
+        "ic"
+      ],
+      [
+        "p",
+        "ec"
+      ],
+      [
+        "p",
+        "res"
+      ],
+      [
+        "t",
+        "s"
+      ],
+      [
+        "t",
+        "w"
+      ],
+      [
+        "t",
+        "en"
+      ],
+      [
+        "t",
+        "an"
+      ],
+      [
+        "t",
+        "ing"
+      ],
+      [
+        "u",
+        "t"
+      ],
+      [
+        "u",
+        "ter"
+      ],
+      [
+        "v",
+        "er"
+      ],
+      [
+        "v",
+        "el"
+      ],
+      [
+        "v",
+        "id"
+      ],
+      [
+        "w",
+        "or"
+      ],
+      [
+        "y",
+        "pic"
+      ],
+      [
+        "Ġ",
+        "("
+      ],
+      [
+        "Ġ",
+        "2"
+      ],
+      [
+        "Ġ",
+        "I"
+      ],
+      [
+        "Ġ",
+        "L"
+      ],
+      [
+        "Ġ",
+        "P"
+      ],
+      [
+        "Ġ",
+        "r"
+      ],
+      [
+        "Ġ",
+        "ex"
+      ],
+      [
+        "Ġ",
+        "NLP"
+      ],
+      [
+        "Ġt",
+        "as"
+      ],
+      [
+        "Ġa",
+        "l"
+      ],
+      [
+        "Ġa",
+        "t"
+      ],
+      [
+        "Ġa",
+        "re"
+      ],
+      [
+        "Ġa",
+        "dded"
+      ],
+      [
+        "an",
+        "d"
+      ],
+      [
+        "at",
+        "ural"
+      ],
+      [
+        "or",
+        "tan"
+      ],
+      [
+        "Ġm",
+        "achine"
+      ],
+      [
+        "Ġm",
+        "ak"
+      ],
+      [
+        "Ġs",
+        "iz"
+      ],
+      [
+        "Ġs",
+        "ent"
+      ],
+      [
+        "Ġs",
+        "amp"
+      ],
+      [
+        "Ġs",
+        "equ"
+      ],
+      [
+        "Ġs",
+        "mal"
+      ],
+      [
+        "al",
+        "le"
+      ],
+      [
+        "le",
+        "vel"
+      ],
+      [
+        "Ġof",
+        "ten"
+      ],
+      [
+        "Ġb",
+        "ro"
+      ],
+      [
+        "Ġp",
+        "art"
+      ],
+      [
+        "ou",
+        "s"
+      ],
+      [
+        "ou",
+        "gh"
+      ],
+      [
+        "Ġc",
+        "omp"
+      ],
+      [
+        "Ġf",
+        "ro"
+      ],
+      [
+        "Ġf",
+        "am"
+      ],
+      [
+        "Ġf",
+        "oc"
+      ],
+      [
+        "Ġin",
+        "clu"
+      ],
+      [
+        "ĠT",
+        "his"
+      ],
+      [
+        "ĠT",
+        "hes"
+      ],
+      [
+        "ĠT",
+        "oo"
+      ],
+      [
+        "Ġu",
+        "s"
+      ],
+      [
+        "Ġu",
+        "se"
+      ],
+      [
+        "Ġbe",
+        "tw"
+      ],
+      [
+        "Ġd",
+        "i"
+      ],
+      [
+        "od",
+        "ing"
+      ],
+      [
+        "enc",
+        "es"
+      ],
+      [
+        "Ġtokeniz",
+        "ation"
+      ],
+      [
+        "as",
+        "ed"
+      ],
+      [
+        "Ġl",
+        "ar"
+      ],
+      [
+        "Ġpro",
+        "gr"
+      ],
+      [
+        "Ġpro",
+        "vid"
+      ],
+      [
+        "Ġh",
+        "um"
+      ],
+      [
+        "Ġh",
+        "ave"
+      ],
+      [
+        "ec",
+        "t"
+      ],
+      [
+        "il",
+        "lions"
+      ],
+      [
+        "Ġn",
+        "et"
+      ],
+      [
+        "Ġn",
+        "eural"
+      ],
+      [
+        "Ġre",
+        "pres"
+      ],
+      [
+        "Ġun",
+        "kn"
+      ],
+      [
+        "00",
+        "0"
+      ],
+      [
+        "su",
+        "pervised"
+      ],
+      [
+        "us",
+        "es"
+      ],
+      [
+        "ĠB",
+        "BPE"
+      ],
+      [
+        "Ġimp",
+        "ortan"
+      ],
+      [
+        "pl",
+        "ic"
+      ],
+      [
+        "Ġen",
+        "su"
+      ],
+      [
+        "Ġhe",
+        "lp"
+      ],
+      [
+        "Ġmu",
+        "ch"
+      ],
+      [
+        "Ġman",
+        "y"
+      ],
+      [
+        "Ġsu",
+        "b"
+      ],
+      [
+        "Ġcorp",
+        "us"
+      ],
+      [
+        "ell",
+        "igence"
+      ],
+      [
+        "Ġint",
+        "elligence"
+      ],
+      [
+        "Ġwith",
+        "out"
+      ],
+      [
+        "Ġtrans",
+        "form"
+      ],
+      [
+        "Ġvocabular",
+        "y"
+      ],
+      [
+        "wor",
+        "ks"
+      ],
+      [
+        "ypic",
+        "al"
+      ],
+      [
+        "Ġtas",
+        "ks"
+      ],
+      [
+        "Ġsamp",
+        "le"
+      ],
+      [
+        "Ġsmal",
+        "l"
+      ],
+      [
+        "Ġcomp",
+        "uter"
+      ],
+      [
+        "Ġfro",
+        "m"
+      ],
+      [
+        "Ġfoc",
+        "uses"
+      ],
+      [
+        "ĠThes",
+        "e"
+      ],
+      [
+        "Ġbetw",
+        "een"
+      ],
+      [
+        "Ġprogr",
+        "am"
+      ],
+      [
+        "Ġprovid",
+        "es"
+      ],
+      [
+        "Ġnet",
+        "works"
+      ],
+      [
+        "Ġrepres",
+        "ent"
+      ],
+      [
+        "Ġunkn",
+        "own"
+      ],
+      [
+        "Ġimportan",
+        "t"
+      ],
+      [
+        "Ġtransform",
+        "er"
+      ]
+    ]
+  }
+}

models/demo_tokenizer/vocab.json

@@ -0,0 +1 @@
+{"<pad>":0,"<unk>":1,"<s>":2,"</s>":3,"<mask>":4,"!":5,"\"":6,"#":7,"$":8,"%":9,"&":10,"'":11,"(":12,")":13,"*":14,"+":15,",":16,"-":17,".":18,"/":19,"0":20,"1":21,"2":22,"3":23,"4":24,"5":25,"6":26,"7":27,"8":28,"9":29,":":30,";":31,"<":32,"=":33,">":34,"?":35,"@":36,"A":37,"B":38,"C":39,"D":40,"E":41,"F":42,"G":43,"H":44,"I":45,"J":46,"K":47,"L":48,"M":49,"N":50,"O":51,"P":52,"Q":53,"R":54,"S":55,"T":56,"U":57,"V":58,"W":59,"X":60,"Y":61,"Z":62,"[":63,"\\":64,"]":65,"^":66,"_":67,"`":68,"a":69,"b":70,"c":71,"d":72,"e":73,"f":74,"g":75,"h":76,"i":77,"j":78,"k":79,"l":80,"m":81,"n":82,"o":83,"p":84,"q":85,"r":86,"s":87,"t":88,"u":89,"v":90,"w":91,"x":92,"y":93,"z":94,"{":95,"|":96,"}":97,"~":98,"¡":99,"¢":100,"£":101,"¤":102,"¥":103,"¦":104,"§":105,"¨":106,"©":107,"ª":108,"«":109,"¬":110,"®":111,"¯":112,"°":113,"±":114,"²":115,"³":116,"´":117,"µ":118,"¶":119,"·":120,"¸":121,"¹":122,"º":123,"»":124,"¼":125,"½":126,"¾":127,"¿":128,"À":129,"Á":130,"Â":131,"Ã":132,"Ä":133,"Å":134,"Æ":135,"Ç":136,"È":137,"É":138,"Ê":139,"Ë":140,"Ì":141,"Í":142,"Î":143,"Ï":144,"Ð":145,"Ñ":146,"Ò":147,"Ó":148,"Ô":149,"Õ":150,"Ö":151,"×":152,"Ø":153,"Ù":154,"Ú":155,"Û":156,"Ü":157,"Ý":158,"Þ":159,"ß":160,"à":161,"á":162,"â":163,"ã":164,"ä":165,"å":166,"æ":167,"ç":168,"è":169,"é":170,"ê":171,"ë":172,"ì":173,"í":174,"î":175,"ï":176,"ð":177,"ñ":178,"ò":179,"ó":180,"ô":181,"õ":182,"ö":183,"÷":184,"ø":185,"ù":186,"ú":187,"û":188,"ü":189,"ý":190,"þ":191,"ÿ":192,"Ā":193,"ā":194,"Ă":195,"ă":196,"Ą":197,"ą":198,"Ć":199,"ć":200,"Ĉ":201,"ĉ":202,"Ċ":203,"ċ":204,"Č":205,"č":206,"Ď":207,"ď":208,"Đ":209,"đ":210,"Ē":211,"ē":212,"Ĕ":213,"ĕ":214,"Ė":215,"ė":216,"Ę":217,"ę":218,"Ě":219,"ě":220,"Ĝ":221,"ĝ":222,"Ğ":223,"ğ":224,"Ġ":225,"ġ":226,"Ģ":227,"ģ":228,"Ĥ":229,"ĥ":230,"Ħ":231,"ħ":232,"Ĩ":233,"ĩ":234,"Ī":235,"ī":236,"Ĭ":237,"ĭ":238,"Į":239,"į":240,"İ":241,"ı":242,"IJ":243,"ij":244,"Ĵ":245,"ĵ":246,"Ķ":247,"ķ":248,"ĸ":249,"Ĺ":250,"ĺ":251,"Ļ":252,"ļ":253,"Ľ":254,"ľ":255,"Ŀ":256,"ŀ":257,"Ł":258,"ł":259,"Ń":260,"Ġt":261,"in":262,"en":263,"Ġa":264,"er":265,"es":266,"Ġo":267,"Ġto":268,"an":269,"ing":270,"ar":271,"at":272,"he":273,"is":274,"or":275,"Ġm":276,"Ġs":277,"al":278,"ken":279,"on":280,"Ġtoken":281,"le":282,"Ġof":283,"Ġb":284,"Ġp":285,"it":286,"ou":287,"Ġc":288,"Ġf":289,"Ġan":290,"ic":291,"iz":292,"ion":293,"ro":294,"ed":295,"el":296,"re":297,"Ġin":298,"Ġthe":299,"Ġis":300,"Ġand":301,"ĠT":302,"Ġu":303,"Ġbe":304,"Ġd":305,"Ġth":306,"ac":307,"mp":308,"od":309,"Ġw":310,"Ġle":311,"enc":312,"ent":313,"Ġtokeniz":314,"as":315,"ex":316,"ter":317,"Ġl":318,"arn":319,"Ġtokens":320,"Ġpro":321,"qu":322,"ul":323,"Ġh":324,"Ġv":325,"ation":326,"Ġthat":327,"Ġlearn":328,"ain":329,"ces":330,"ec":331,"fic":332,"ge":333,"il":334,"res":335,"ua":336,"you":337,"Ġn":338,"Ġre":339,"Ġyou":340,"Ġtr":341,"Ġtex":342,"Ġun":343,"cess":344,"Ġtext":345,"00":346,"am":347,"ds":348,"em":349,"et":350,"gua":351,"ific":352,"oc":353,"per":354,"st":355,"ur":356,"ve":357,"Ġon":358,"angua":359,"ords":360,"Ġmod":361,"ence":362,"Ġtokenizer":363,"ular":364,"Ġlearning":365,"aining":366,"ab":367,"ch":368,"for":369,"hin":370,"his":371,"ial":372,"imp":373,"ks":374,"se":375,"su":376,"us":377,"ĠB":378,"Ġ[":379,"Ġimp":380,"Ġar":381,"Ġor":382,"ans":383,"ate":384,"ith":385,"Ġcan":386,"Ġfor":387,"achin":388,"Ġwords":389,"Ġlangua":390,"Ġprocess":391,"Ġyour":392,"ural":393,"Ġmodel":394,"achine":395,"LP":396,"NLP":397,"PE":398,"The":399,"],":400,"ad":401,"ded":402,"ev":403,"gh":404,"id":405,"im":406,"ld":407,"lp":408,"lu":409,"ly":410,"pl":411,"te":412,"tion":413,"tific":414,"um":415,"vis":416,"wn":417,"yte":418,"Ġen":419,"Ġhe":420,"Ġit":421,"Ġqu":422,"art":423,"ata":424,"orp":425,"Ġmu":426,"Ġman":427,"Ġsu":428,"ont":429,"ity":430,"out":431,"Ġcorp":432,"Ġcont":433,"ions":434,"ell":435,"Ġint":436,"ĠThe":437,"Ġdata":438,"Ġwith":439,"Ġval":440,"Ġvoc":441,"Ġtraining":442,"Ġtrans":443,"pervis":444,"abular":445,"form":446,"Ġartific":447,"Ġlanguage":448,"Ġmodels":449,"Ġvocabular":450,"pervised":451,"Ġartificial":452,"BPE":453,"Byte":454,"Machine":455,"This":456,"ak":457,"amp":458,"ave":459,"clu":460,"ding":461,"dded":462,"een":463,"equ":464,"eural":465,"gr":466,"gence":467,"her":468,"hes":469,"ies":470,"ive":471,"igence":472,"kn":473,"lions":474,"ms":475,"mal":476,"om":477,"oo":478,"op":479,"omp":480,"own":481,"pp":482,"pic":483,"pec":484,"pres":485,"ts":486,"tw":487,"ten":488,"tan":489,"ting":490,"ut":491,"uter":492,"ver":493,"vel":494,"vid":495,"wor":496,"ypic":497,"Ġ(":498,"Ġ2":499,"ĠI":500,"ĠL":501,"ĠP":502,"Ġr":503,"Ġex":504,"ĠNLP":505,"Ġtas":506,"Ġal":507,"Ġat":508,"Ġare":509,"Ġadded":510,"and":511,"atural":512,"ortan":513,"Ġmachine":514,"Ġmak":515,"Ġsiz":516,"Ġsent":517,"Ġsamp":518,"Ġsequ":519,"Ġsmal":520,"alle":521,"level":522,"Ġoften":523,"Ġbro":524,"Ġpart":525,"ous":526,"ough":527,"Ġcomp":528,"Ġfro":529,"Ġfam":530,"Ġfoc":531,"Ġinclu":532,"ĠThis":533,"ĠThes":534,"ĠToo":535,"Ġus":536,"Ġuse":537,"Ġbetw":538,"Ġdi":539,"oding":540,"ences":541,"Ġtokenization":542,"ased":543,"Ġlar":544,"Ġprogr":545,"Ġprovid":546,"Ġhum":547,"Ġhave":548,"ect":549,"illions":550,"Ġnet":551,"Ġneural":552,"Ġrepres":553,"Ġunkn":554,"000":555,"supervised":556,"uses":557,"ĠBBPE":558,"Ġimportan":559,"plic":560,"Ġensu":561,"Ġhelp":562,"Ġmuch":563,"Ġmany":564,"Ġsub":565,"Ġcorpus":566,"elligence":567,"Ġintelligence":568,"Ġwithout":569,"Ġtransform":570,"Ġvocabulary":571,"works":572,"ypical":573,"Ġtasks":574,"Ġsample":575,"Ġsmall":576,"Ġcomputer":577,"Ġfrom":578,"Ġfocuses":579,"ĠThese":580,"Ġbetween":581,"Ġprogram":582,"Ġprovides":583,"Ġnetworks":584,"Ġrepresent":585,"Ġunknown":586,"Ġimportant":587,"Ġtransformer":588}

requirements.txt

@@ -0,0 +1 @@
+tokenizers>=0.15.0

scripts/bbpe_trainer.py

@@ -0,0 +1,321 @@
+"""
+Byte-Level BPE Tokenizer Training Pipeline
+
+This module provides a comprehensive architecture for training Byte-Level BPE (BBPE) tokenizers
+using Hugging Face's `tokenizers` library. It includes data preprocessing, training configuration,
+and model serialization utilities.
+"""
+
+import os
+import json
+from pathlib import Path
+from typing import List, Optional, Union
+from dataclasses import dataclass, field, asdict
+from tokenizers import ByteLevelBPETokenizer, Tokenizer
+
+
+@dataclass
+class BBPEConfig:
+    """Configuration class for BBPE tokenizer training."""
+    
+    # Vocabulary settings
+    vocab_size: int = 30000
+    min_frequency: int = 2
+    
+    # Special tokens
+    special_tokens: List[str] = field(default_factory=lambda: [
+        "<pad>",
+        "<unk>",
+        "<s>",
+        "</s>",
+        "<mask>"
+    ])
+    
+    # Byte-level settings
+    lowercase: bool = False
+    add_prefix_space: bool = True
+    trim_offsets: bool = False
+    
+    # Training settings
+    show_progress: bool = True
+    initial_alphabet: List[str] = field(default_factory=list)
+    
+    # Paths
+    data_dir: str = "data"
+    model_save_dir: str = "models"
+    model_name: str = "bbpe_tokenizer"
+    
+    def to_dict(self) -> dict:
+        """Convert config to dictionary."""
+        return asdict(self)
+    
+    def save(self, path: str):
+        """Save configuration to JSON file."""
+        with open(path, 'w', encoding='utf-8') as f:
+            json.dump(self.to_dict(), f, indent=2, ensure_ascii=False)
+    
+    @classmethod
+    def load(cls, path: str) -> 'BBPEConfig':
+        """Load configuration from JSON file."""
+        with open(path, 'r', encoding='utf-8') as f:
+            config_dict = json.load(f)
+        return cls(**config_dict)
+
+
+class BBPETrainer:
+    """
+    End-to-end trainer for Byte-Level BPE tokenizers.
+    
+    This class handles the complete training pipeline including:
+    - Data loading and preprocessing
+    - Tokenizer initialization with byte-level encoding
+    - BPE training with configurable parameters
+    - Model saving and loading
+    """
+    
+    def __init__(self, config: Optional[BBPEConfig] = None):
+        """
+        Initialize the BBPE trainer.
+        
+        Args:
+            config: BBPEConfig instance. If None, default config is used.
+        """
+        self.config = config or BBPEConfig()
+        self.tokenizer: Optional[ByteLevelBPETokenizer] = None
+        self._setup_directories()
+    
+    def _setup_directories(self):
+        """Create necessary directories for data and models."""
+        Path(self.config.data_dir).mkdir(parents=True, exist_ok=True)
+        Path(self.config.model_save_dir).mkdir(parents=True, exist_ok=True)
+    
+    def initialize_tokenizer(self) -> ByteLevelBPETokenizer:
+        """
+        Initialize a new ByteLevelBPETokenizer with byte-level encoding.
+        
+        Returns:
+            Initialized ByteLevelBPETokenizer instance
+        """
+        tokenizer = ByteLevelBPETokenizer(
+            add_prefix_space=self.config.add_prefix_space,
+            trim_offsets=self.config.trim_offsets,
+            lowercase=self.config.lowercase,
+        )
+        self.tokenizer = tokenizer
+        return tokenizer
+    
+    def get_training_files(self) -> List[str]:
+        """
+        Get list of text files for training from the data directory.
+        
+        Returns:
+            List of file paths to text files
+        """
+        data_path = Path(self.config.data_dir)
+        text_files = []
+        
+        # Support multiple text file extensions
+        extensions = ['.txt', '.jsonl', '.json']
+        
+        for ext in extensions:
+            text_files.extend(list(data_path.glob(f'*{ext}')))
+        
+        if not text_files:
+            raise FileNotFoundError(
+                f"No training files found in {data_path}. "
+                f"Please add .txt, .json, or .jsonl files to this directory."
+            )
+        
+        return [str(f) for f in text_files]
+    
+    def train(self, 
+              files: Optional[List[str]] = None,
+              config_override: Optional[dict] = None) -> ByteLevelBPETokenizer:
+        """
+        Train the BBPE tokenizer on the provided files.
+        
+        Args:
+            files: List of file paths to train on. If None, uses files from data_dir.
+            config_override: Optional dictionary to override config parameters.
+            
+        Returns:
+            Trained ByteLevelBPETokenizer instance
+        """
+        # Apply config overrides if provided
+        if config_override:
+            for key, value in config_override.items():
+                if hasattr(self.config, key):
+                    setattr(self.config, key, value)
+        
+        # Initialize tokenizer if not already done
+        if self.tokenizer is None:
+            self.initialize_tokenizer()
+        
+        # Get training files
+        if files is None:
+            files = self.get_training_files()
+        
+        print(f"Training on {len(files)} file(s)...")
+        for f in files:
+            print(f"  - {f}")
+        
+        # Train the tokenizer using the new API (tokenizers >= 0.15)
+        print("\nStarting training...")
+        self.tokenizer.train(
+            files=files,
+            vocab_size=self.config.vocab_size,
+            min_frequency=self.config.min_frequency,
+            special_tokens=self.config.special_tokens,
+            show_progress=self.config.show_progress,
+        )
+        print("Training completed!")
+        
+        # Print vocabulary statistics
+        vocab_size = self.tokenizer.get_vocab_size()
+        print(f"\nVocabulary size: {vocab_size}")
+        print(f"Special tokens: {self.config.special_tokens}")
+        
+        return self.tokenizer
+    
+    def save(self, model_name: Optional[str] = None) -> str:
+        """
+        Save the trained tokenizer to disk.
+        
+        Args:
+            model_name: Name for the saved model. If None, uses config.model_name.
+            
+        Returns:
+            Path to the saved model directory
+        """
+        if self.tokenizer is None:
+            raise ValueError("No tokenizer to save. Please train first.")
+        
+        name = model_name or self.config.model_name
+        save_path = Path(self.config.model_save_dir) / name
+        save_path.mkdir(parents=True, exist_ok=True)
+        
+        # Save tokenizer files
+        self.tokenizer.save_model(str(save_path))
+        
+        # Save configuration
+        config_path = save_path / "config.json"
+        self.config.save(str(config_path))
+        
+        # Save tokenizer.json (full tokenizer state)
+        tokenizer_json_path = save_path / "tokenizer.json"
+        self.tokenizer.save(str(tokenizer_json_path))
+        
+        print(f"\nTokenizer saved to: {save_path}")
+        print(f"  - vocab.json")
+        print(f"  - merges.txt")
+        print(f"  - config.json")
+        print(f"  - tokenizer.json")
+        
+        return str(save_path)
+    
+    def load(self, model_path: str) -> ByteLevelBPETokenizer:
+        """
+        Load a pre-trained tokenizer from disk.
+        
+        Args:
+            model_path: Path to the directory containing tokenizer files.
+            
+        Returns:
+            Loaded ByteLevelBPETokenizer instance
+        """
+        model_path = Path(model_path)
+        
+        if not model_path.exists():
+            raise FileNotFoundError(f"Model path does not exist: {model_path}")
+        
+        # Try to load tokenizer.json first (preferred method for tokenizers >= 0.15)
+        tokenizer_json = model_path / "tokenizer.json"
+        if tokenizer_json.exists():
+            # Use the generic Tokenizer class to load the full tokenizer state
+            base_tokenizer = Tokenizer.from_file(str(tokenizer_json))
+            # Wrap it as ByteLevelBPETokenizer for consistent API
+            self.tokenizer = ByteLevelBPETokenizer(
+                add_prefix_space=self.config.add_prefix_space,
+                trim_offsets=self.config.trim_offsets,
+                lowercase=self.config.lowercase,
+            )
+            # Copy the vocabulary and merges from the loaded tokenizer
+            self.tokenizer = base_tokenizer
+        else:
+            # Fall back to loading vocab.json and merges.txt
+            vocab_file = model_path / "vocab.json"
+            merges_file = model_path / "merges.txt"
+            
+            if not vocab_file.exists() or not merges_file.exists():
+                raise FileNotFoundError(
+                    f"Required files not found in {model_path}. "
+                    f"Need either tokenizer.json or both vocab.json and merges.txt"
+                )
+            
+            self.tokenizer = ByteLevelBPETokenizer.from_file(
+                str(vocab_file), str(merges_file)
+            )
+        
+        # Load config if exists
+        config_file = model_path / "config.json"
+        if config_file.exists():
+            self.config = BBPEConfig.load(str(config_file))
+        
+        print(f"Tokenizer loaded from: {model_path}")
+        return self.tokenizer
+    
+    def encode(self, text: str, **kwargs) -> List[int]:
+        """Encode text to token IDs."""
+        if self.tokenizer is None:
+            raise ValueError("No tokenizer loaded. Please train or load first.")
+        return self.tokenizer.encode(text, **kwargs).ids
+    
+    def decode(self, ids: List[int], **kwargs) -> str:
+        """Decode token IDs to text."""
+        if self.tokenizer is None:
+            raise ValueError("No tokenizer loaded. Please train or load first.")
+        return self.tokenizer.decode(ids, **kwargs)
+    
+    def tokenize(self, text: str) -> List[str]:
+        """Tokenize text to token strings."""
+        if self.tokenizer is None:
+            raise ValueError("No tokenizer loaded. Please train or load first.")
+        return self.tokenizer.encode(text).tokens
+
+
+def main():
+    """Example usage of the BBPE trainer."""
+    # Create configuration
+    config = BBPEConfig(
+        vocab_size=30000,
+        min_frequency=2,
+        special_tokens=["<pad>", "<unk>", "<s>", "</s>", "<mask>"],
+        data_dir="data",
+        model_save_dir="models",
+        model_name="my_bbpe_tokenizer",
+    )
+    
+    # Initialize trainer
+    trainer = BBPETrainer(config)
+    
+    # Train the tokenizer
+    trainer.train()
+    
+    # Save the tokenizer
+    save_path = trainer.save()
+    
+    # Test encoding/decoding
+    test_text = "Hello, world! This is a test of the BBPE tokenizer."
+    encoded = trainer.encode(test_text)
+    decoded = trainer.decode(encoded)
+    tokens = trainer.tokenize(test_text)
+    
+    print(f"\nTest encoding:")
+    print(f"  Input: {test_text}")
+    print(f"  Tokens: {tokens}")
+    print(f"  IDs: {encoded}")
+    print(f"  Decoded: {decoded}")
+
+
+if __name__ == "__main__":
+    main()

scripts/example_usage.py

@@ -0,0 +1,67 @@
+"""
+EthioBBPE Example Usage
+
+Demonstrates how to use the trained EthioBBPE tokenizer for Ethiopian languages.
+"""
+
+from tokenizers import Tokenizer
+import os
+
+def load_tokenizer(model_path="models/EthioBBPE/tokenizer.json"):
+    """Load the trained EthioBBPE tokenizer."""
+    if not os.path.exists(model_path):
+        # Try demo model
+        demo_path = "models/demo_tokenizer/tokenizer.json"
+        if os.path.exists(demo_path):
+            print(f"⚠️  EthioBBPE model not found, using demo model instead.")
+            model_path = demo_path
+        else:
+            raise FileNotFoundError(
+                f"No tokenizer found at {model_path}. Please train a model first."
+            )
+    
+    return Tokenizer.from_file(model_path)
+
+
+def main():
+    print("🇪🇹 EthioBBPE Tokenizer - Example Usage\n")
+    print("=" * 50)
+    
+    # Load tokenizer
+    tokenizer = load_tokenizer()
+    print(f"✅ Loaded tokenizer from: {tokenizer.model_filename}\n")
+    
+    # Test texts in multiple Ethiopian languages
+    test_texts = [
+        ("Amharic", "ሰላም! እንዴት ነህ? የኢትዮጵያ ህዝብ በጣም ተቀራራቢ ነው።"),
+        ("Oromo", "Akkam! Akkam jirta? Ummanni Itoophiyaa baay'ee wal-qabaataa dha."),
+        ("Tigrinya", "ሰላም! ከመይ ኣለኻ? ህዝቢ ኢትዮጵያ ኣዝዩ ሓደ እዩ።"),
+        ("English", "Hello! How are you? The people of Ethiopia are very united."),
+        ("Mixed", "ሰላም Hello! እንዴት ነህ? How are you? 🇪🇹"),
+    ]
+    
+    for lang_name, text in test_texts:
+        print(f"\n--- {lang_name} ---")
+        print(f"Original: {text}")
+        
+        # Encode
+        encoded = tokenizer.encode(text)
+        print(f"Tokens ({len(encoded.tokens)}): {encoded.tokens[:20]}{'...' if len(encoded.tokens) > 20 else ''}")
+        print(f"IDs ({len(encoded.ids)}): {encoded.ids[:20]}{'...' if len(encoded.ids) > 20 else ''}")
+        
+        # Decode
+        decoded = tokenizer.decode(encoded.ids)
+        print(f"Decoded: {decoded}")
+        
+        # Verify round-trip
+        match = "✅" if decoded == text else "⚠️"
+        print(f"Round-trip: {match} {'Perfect match!' if decoded == text else 'Minor differences'}")
+    
+    print("\n" + "=" * 50)
+    print("✨ Example usage complete!")
+    print("\nTo train your own EthioBBPE tokenizer:")
+    print("  python scripts/train_tokenizer.py --data_dir ./data --model_name EthioBBPE")
+
+
+if __name__ == "__main__":
+    main()

scripts/train_tokenizer.py

@@ -0,0 +1,205 @@
+#!/usr/bin/env python3
+"""
+Command-line interface for training BBPE tokenizers.
+
+Usage:
+    python train_tokenizer.py --data_dir ./data --vocab_size 30000 --model_name my_tokenizer
+"""
+
+import argparse
+import sys
+from pathlib import Path
+
+# Add parent directory to path for imports
+sys.path.insert(0, str(Path(__file__).parent))
+
+from bbpe_trainer import BBPETrainer, BBPEConfig
+
+
+def parse_args():
+    """Parse command-line arguments."""
+    parser = argparse.ArgumentParser(
+        description="Train a Byte-Level BPE (BBPE) tokenizer",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    
+    # Data arguments
+    parser.add_argument(
+        "--data_dir",
+        type=str,
+        default="data",
+        help="Directory containing training text files (.txt, .json, .jsonl)",
+    )
+    
+    parser.add_argument(
+        "--files",
+        type=str,
+        nargs="+",
+        default=None,
+        help="Specific files to train on (overrides data_dir)",
+    )
+    
+    # Model arguments
+    parser.add_argument(
+        "--vocab_size",
+        type=int,
+        default=30000,
+        help="Target vocabulary size",
+    )
+    
+    parser.add_argument(
+        "--min_frequency",
+        type=int,
+        default=2,
+        help="Minimum frequency for tokens to be included in vocabulary",
+    )
+    
+    parser.add_argument(
+        "--special_tokens",
+        type=str,
+        nargs="+",
+        default=["<pad>", "<unk>", "<s>", "</s>", "<mask>"],
+        help="Special tokens to add to the vocabulary",
+    )
+    
+    # Training options
+    parser.add_argument(
+        "--lowercase",
+        action="store_true",
+        help="Convert text to lowercase before tokenization",
+    )
+    
+    parser.add_argument(
+        "--no_prefix_space",
+        action="store_true",
+        help="Disable adding prefix space (default: add prefix space)",
+    )
+    
+    parser.add_argument(
+        "--show_progress",
+        action="store_true",
+        default=True,
+        help="Show training progress bar",
+    )
+    
+    parser.add_argument(
+        "--no_progress",
+        action="store_false",
+        dest="show_progress",
+        help="Hide training progress bar",
+    )
+    
+    # Output arguments
+    parser.add_argument(
+        "--model_save_dir",
+        type=str,
+        default="models",
+        help="Directory to save the trained tokenizer",
+    )
+    
+    parser.add_argument(
+        "--model_name",
+        type=str,
+        default="bbpe_tokenizer",
+        help="Name for the saved tokenizer model",
+    )
+    
+    # Config file arguments
+    parser.add_argument(
+        "--config_file",
+        type=str,
+        default=None,
+        help="Path to JSON config file (overrides other arguments)",
+    )
+    
+    parser.add_argument(
+        "--save_config",
+        type=str,
+        default=None,
+        help="Path to save the configuration JSON file",
+    )
+    
+    return parser.parse_args()
+
+
+def main():
+    """Main entry point for CLI training."""
+    args = parse_args()
+    
+    # Load config from file if provided
+    if args.config_file:
+        print(f"Loading configuration from {args.config_file}")
+        config = BBPEConfig.load(args.config_file)
+    else:
+        # Create config from arguments
+        config = BBPEConfig(
+            vocab_size=args.vocab_size,
+            min_frequency=args.min_frequency,
+            special_tokens=args.special_tokens,
+            lowercase=args.lowercase,
+            add_prefix_space=not args.no_prefix_space,
+            show_progress=args.show_progress,
+            data_dir=args.data_dir,
+            model_save_dir=args.model_save_dir,
+            model_name=args.model_name,
+        )
+    
+    # Save config if requested
+    if args.save_config:
+        config.save(args.save_config)
+        print(f"Configuration saved to {args.save_config}")
+    
+    # Initialize trainer
+    trainer = BBPETrainer(config)
+    
+    # Get training files
+    if args.files:
+        print(f"Using specified files: {args.files}")
+        files = args.files
+    else:
+        files = None  # Will use files from data_dir
+    
+    # Train the tokenizer
+    try:
+        trainer.train(files=files)
+    except FileNotFoundError as e:
+        print(f"\nError: {e}")
+        print("\nTo fix this:")
+        print(f"  1. Add your training data to the '{args.data_dir}' directory")
+        print("  2. Supported formats: .txt, .json, .jsonl")
+        print("  3. Or specify files directly with --files flag")
+        sys.exit(1)
+    
+    # Save the tokenizer
+    save_path = trainer.save()
+    
+    # Test the tokenizer
+    print("\n" + "="*60)
+    print("TESTING TOKENIZER")
+    print("="*60)
+    
+    test_texts = [
+        "Hello, world!",
+        "This is a test of the BBPE tokenizer.",
+        "Special characters: @#$%^&*()",
+        "Numbers: 12345 and words mixed together.",
+    ]
+    
+    for text in test_texts:
+        encoded = trainer.encode(text)
+        tokens = trainer.tokenize(text)
+        decoded = trainer.decode(encoded)
+        
+        print(f"\nInput:    {text}")
+        print(f"Tokens:   {tokens}")
+        print(f"IDs:      {encoded[:20]}{'...' if len(encoded) > 20 else ''}")
+        print(f"Decoded:  {decoded}")
+    
+    print("\n" + "="*60)
+    print(f"Tokenizer training complete!")
+    print(f"Model saved to: {save_path}")
+    print("="*60)
+
+
+if __name__ == "__main__":
+    main()