# 📘 GPT-Style 50K Multilingual Tokenizer

A **byte-level BPE tokenizer** trained on a large mixed corpus of **English + Urdu text**, designed for chat-style LLMs with support for structured prompts and multimodal placeholders.

---

## 🚀 Overview

This tokenizer is built using **Byte Pair Encoding (BPE)** with a vocabulary size of **50,000 tokens**, optimized for:

* English language text
* Urdu (Unicode script)
* Chat formatting (system/user/assistant)
* Multimodal placeholders (speech, frames)

It is compatible with Hugging Face Transformers.

---

## ✨ Features

* ⚡ 50,000 vocabulary size
* 🌍 Multilingual support (English + Urdu)
* 💬 Chat-format tokens
* 🧠 Byte-level BPE (robust to unknown characters)
* 🔁 Fully reversible encoding/decoding
* 📦 Hugging Face compatible (`PreTrainedTokenizerFast`)
* 🧩 Special token system for LLM training

---

## 🧠 Special Tokens

### Core Tokens

```
<|pad|> <|unk|> <|bos|> <|eos|>
```

### Chat Tokens

```
<|im_start|> <|im_end|>
<|system|> <|user|> <|assistant|>
```

### Multimodal Tokens

```
<|speech|> <|frame_start|> <|frame_end|>
```

### Reserved Tokens

```
<|reserved_1|> ... <|reserved_80|>
```

---

## 📊 Training Data

The tokenizer was trained on a large-scale mixed dataset:

* English corpora (web text)
* Urdu datasets
* FineWeb samples (filtered parquet files)
* Chat-formatted synthetic data

Total training size: **~3M+ rows**

---

## ⚙️ Training Method

* Algorithm: **Byte Pair Encoding (BPE)**
* Normalization: `NFKC`
* Pre-tokenization: `ByteLevel`
* Decoder: `ByteLevelDecoder`
* Trainer: `BpeTrainer`
* Vocab size: `50,000`

---

## 📦 Installation

```bash
pip install transformers tokenizers huggingface_hub
```

---

## 📥 Loading the Tokenizer

```python
from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained("Humair332/gpt-style-50k-tokenizer")
```

---

## 🧪 Example Usage

### Encoding

```python
text = "Hello world! آپ کیسے ہیں؟"

encoded = tokenizer(text)
print(encoded["input_ids"])
```

---

### Decoding

```python
decoded = tokenizer.decode(encoded["input_ids"])
print(decoded)
```

---

## 💬 Chat Format Example

```text
<|im_start|><|system|>
You are a helpful assistant.
<|im_end|>

<|im_start|><|user|>
ہیلو، آپ کیسے ہیں؟
<|im_end|>

<|im_start|><|assistant|>
```

---

## 📊 Performance Metrics

| Metric          | English | Urdu |
| --------------- | ------- | ---- |
| Chars/Token     | ~5.1    | ~3.9 |
| Tokens/Word     | ~1.0    | ~1.2 |
| Lossless Decode | ✅       | ✅    |

---

## 🧠 Notes on Tokenization Behavior

* English text is highly efficient (near word-level tokens)
* Urdu uses subword + byte-level encoding
* Some Urdu words may be split into multiple sub-tokens
* Tokenization is fully reversible and lossless

---

## 📁 Output Format

The tokenizer outputs:

```python
{
  "input_ids": [...],
  "attention_mask": [...]
}
```

---

## 🔥 Strengths

* Strong English performance
* Good Urdu coverage
* Stable chat formatting
* Robust to unseen characters
* Suitable for LLM pretraining and SFT

---

## ⚠️ Limitations

* Urdu tokens may appear byte-split internally
* Not fully optimized for morphological Urdu compression
* Vocabulary size may be small for very large-scale multilingual use

---

## 📌 Repository

```
Humair332/gpt-style-50k-tokenizer
```

---