---
language:
- am
license: apache-2.0
tags:
- tokenizers
- amharic
- geez
- ethiopic
- biblical-texts
- synaxarium
- byte-level-bpe
datasets:
- Nexuss0781/synaxarium
- Nexuss0781/conon-biblical-am-en
metrics:
- perfect-reconstruction
widget:
- text: "ሰላም ለኢዮብ ዘኢነበበ ከንቶ ።"
---

# 🇪🇹 EthioBBPE - Amharic Biblical Tokenizer

[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Hugging Face](https://img.shields.io/badge/🤗-Hugging_Face-yellow)](https://huggingface.co/Nexuss0781/Ethio-BBPE)
[![GitHub](https://img.shields.io/badge/GitHub-Repo-black?logo=github)](https://github.com/nexuss0781/Ethio_BBPE)
[![PyPI](https://img.shields.io/badge/PyPI-EthioBBPE-blue?logo=pypi)](https://pypi.org/project/EthioBBPE/)
[![Amharic](https://img.shields.io/badge/Language-Amharic-green.svg)](https://en.wikipedia.org/wiki/Amharic)
[![Tokenizer Type](https://img.shields.io/badge/Type-Byte--level_BPE-orange.svg)](https://huggingface.co/docs/tokenizers/index)

A production-ready **Byte-level BPE tokenizer** specifically trained on **Amharic biblical and religious texts**, achieving **perfect reconstruction** of complex Ge'ez script, ancient punctuation, and liturgical content.

## ✨ Features

- ✅ **Perfect Reconstruction**: 100% accuracy on all test samples including ancient Ge'ez punctuation
- ✅ **Specialized Vocabulary**: Trained on 61,769 lines of Amharic biblical texts (Synaxarium + Canon Bible)
- ✅ **Compressed Storage**: Gzip compression (level 9) reduces model size by **89.8%** (1.3MB → 136KB)
- ✅ **Production Ready**: Checkpointing, metrics tracking, and comprehensive error handling
- ✅ **Ge'ez Script Support**: Full support for Ethiopic characters, numerals, and liturgical punctuation marks
- ✅ **Easy Installation**: `pip install EthioBBPE` for immediate use

## 📊 Training Data

| Dataset | Source | Texts | Description |
|---------|--------|-------|-------------|
| **Synaxarium** | [Nexuss0781/synaxarium](https://huggingface.co/datasets/Nexuss0781/synaxarium) | 366 | Daily synaxarium readings in Amharic |
| **Canon Biblical** | [Nexuss0781/conon-biblical-am-en](https://huggingface.co/datasets/Nexuss0781/conon-biblical-am-en) | 61,403 | Amharic-English biblical texts |
| **Total** | - | **61,769** | **15.43 MB** combined corpus |

### Training Configuration

```json
{
  "vocab_size": 16000,
  "min_frequency": 2,
  "special_tokens": ["<pad>", "<unk>", "<s>", "</s>"],
  "lowercase": false,
  "compression": "gzip (level 9)",
  "checkpointing": true
}
```

## 🚀 Quick Start

### Installation

```bash
# Install from PyPI
pip install EthioBBPE

# Or install from source
git clone https://github.com/nexuss0781/Ethio_BBPE.git
cd Ethio_BBPE
pip install -e .
```

### Basic Usage

```python
from ethiobbpe import EthioBBPE

# Load pretrained tokenizer
tokenizer = EthioBBPE.from_pretrained("Nexuss0781/Ethio-BBPE")

# Encode Amharic text
text = "ሰላም ለኢዮብ ዘኢነበበ ከንቶ ።"
encoded = tokenizer.encode(text)

print(f"Tokens: {encoded.tokens}")
print(f"IDs: {encoded.ids}")

# Decode back to text
decoded = tokenizer.decode(encoded.ids)
print(f"Decoded: {decoded}")
# Output: ሰላም ለኢዮብ ዘኢነበበ ከንቶ ። (Perfect reconstruction!)
```

### Advanced Usage

```python
from ethiobbpe import EthioBBPE

# Load with custom configuration
tokenizer = EthioBBPE.from_pretrained(
    "Nexuss0781/Ethio-BBPE",
    use_compressed=True  # Use compressed vocab (faster loading)
)

# Batch encoding
texts = [
    "በመዠመሪያ፡እግዚአብሔር፡ሰማይንና፡ምድርን፡ፈጠረ።",
    "ወደ ቍስጥንጥንያ አገርም በደረሰች ጊዜ",
    "፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠"  # Ancient Ge'ez punctuation
]

encodings = tokenizer.encode_batch(texts)
for i, enc in enumerate(encodings):
    print(f"Text {i}: {len(enc.ids)} tokens")

# Perfect reconstruction test
for text in texts:
    encoded = tokenizer.encode(text)
    decoded = tokenizer.decode(encoded.ids)
    assert text == decoded, f"Mismatch: {text} != {decoded}"
    
print("✅ All texts reconstructed perfectly!")
```

## 📈 Performance Metrics

| Metric | Value |
|--------|-------|
| **Vocabulary Size** | 16,000 tokens |
| **Training Time** | ~17 seconds |
| **Compression Ratio** | 89.8% (gzip level 9) |
| **Reconstruction Accuracy** | 100% ✅ |
| **Model Size (compressed)** | 136 KB |
| **Model Size (uncompressed)** | 1.3 MB |

### Test Results

```python
# Test 1: Ge'ez Punctuation
text = "፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠"
encoded = tokenizer.encode(text)
decoded = tokenizer.decode(encoded.ids)
assert text == decoded  # ✅ PASS (1 token)

# Test 2: Synaxarium Text
text = "ሰላም ለኢዮብ ዘኢነበበ ከንቶ ። አመ አኀዞ አበቅ ወአመ አህጎለ ጥሪቶ ።"
encoded = tokenizer.encode(text)
decoded = tokenizer.decode(encoded.ids)
assert text == decoded  # ✅ PASS (66 tokens)

# Test 3: Biblical Narrative
text = "ወደ ቍስጥንጥንያ አገርም በደረሰች ጊዜ ያቺ ሴት ወደ ንጉሡ ሒዳ..."
encoded = tokenizer.encode(text)
decoded = tokenizer.decode(encoded.ids)
assert text == decoded  # ✅ PASS (82 tokens)
```

## 🛠️ Training Your Own Tokenizer

The repository includes production-ready training scripts with advanced features:

```bash
# Basic training
python scripts/train_tokenizer.py --data_dir ./data --vocab_size 16000

# With compression and checkpointing
python scripts/train_tokenizer.py \
    --data_dir ./data \
    --vocab_size 16000 \
    --model_name MyTokenizer \
    --compression_format gzip \
    --compression_level 9 \
    --use_checkpoint \
    --max_checkpoints 5

# Custom special tokens
python scripts/train_tokenizer.py \
    --data_dir ./data \
    --special_tokens <pad> <unk> <s> </s> <mask>
```

### Advanced Features

- **Checkpointing**: Automatically saves training progress with SHA256 integrity verification
- **Multi-format Compression**: Supports gzip, bz2, and lzma/xz with configurable levels (1-9)
- **Quantization**: Optional 8-bit or 4-bit quantization for deployment optimization
- **Metrics Tracking**: Comprehensive training statistics saved to JSON
- **Parallel Processing**: Multi-threaded training for faster convergence

## 📁 Repository Structure

```
Ethio_BBPE/
├── models/
│   └── EthioBBPE/
│       ├── tokenizer.json          # Standard tokenizer format
│       ├── vocab.json.gz           # Compressed vocabulary
│       ├── config.json             # Model configuration
│       ├── training_metrics.json   # Training statistics
│       └── README.md               # Model card
├── scripts/
│   ├── bbpe_trainer.py             # Core trainer with advanced features
│   ├── train_tokenizer.py          # CLI interface
│   ├── prepare_data.py             # Data preparation utilities
│   └── example_usage.py            # Usage examples
├── src/ethiobbpe/
│   ├── __init__.py                 # Package initialization
│   └── tokenizer.py                # Clean API interface
├── pyproject.toml                  # PyPI package configuration
├── requirements.txt                # Dependencies
├── LICENSE                         # Apache 2.0
└── README.md                       # This file
```

## 🔗 Links

- **Hugging Face Model**: https://huggingface.co/Nexuss0781/Ethio-BBPE
- **GitHub Repository**: https://github.com/nexuss0781/Ethio_BBPE
- **PyPI Package**: https://pypi.org/project/EthioBBPE/
- **Synaxarium Dataset**: https://huggingface.co/datasets/Nexuss0781/synaxarium
- **Canon Biblical Dataset**: https://huggingface.co/datasets/Nexuss0781/conon-biblical-am-en

## 📄 License

This project is licensed under the **Apache License 2.0**. See the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

- Training data from [Nexuss0781/synaxarium](https://huggingface.co/datasets/Nexuss0781/synaxarium) and [Nexuss0781/conon-biblical-am-en](https://huggingface.co/datasets/Nexuss0781/conon-biblical-am-en)
- Built with [Hugging Face Tokenizers](https://huggingface.co/docs/tokenizers/index)
- Inspired by the rich tradition of Ethiopian biblical literature

## 📧 Contact

For questions or issues, please open an issue on [GitHub](https://github.com/nexuss0781/Ethio_BBPE/issues).