README.md

---
language:
- multilingual
tags:
- ethiobbpe
- bpe
- tokenizer
- byte-level
license: apache-2.0
datasets:
- user-provided
---

# EthioBBPE Tokenizer

This is a production-ready Byte-Level BPE tokenizer with advanced features for deployment.

## Features
- **Byte-Level**: Handles any Unicode character without <UNK>.
- **Multi-format Compression**: Supports gzip, bz2, and lzma compression.
- **Checkpointing**: Built-in safety checkpoints with metadata tracking.
- **Quantization**: Optional 8-bit/4-bit quantization for efficient deployment.
- **Training Metrics**: Comprehensive metrics tracking and logging.
- **Automatic Backup**: Checkpoint rotation to manage disk space.

## Usage

### Transformers
```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("EthioBBPE_AmharicBible")
```

### Tokenizers Library
```python
from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")
```

### Loading Compressed Vocab
```python
import gzip
import json

# Load compressed vocabulary
with gzip.open("vocab.json.gz", 'rt', encoding='utf-8') as f:
    vocab = json.load(f)
```

## Training Configuration
```json
{
  "vocab_size": 16000,
  "min_frequency": 2,
  "show_progress": true,
  "special_tokens": [
    "<pad>",
    "<unk>",
    "<s>",
    "</s>",
    "<mask>"
  ],
  "lowercase": false,
  "dropout": null,
  "data_dir": "./data",
  "model_save_dir": "models",
  "model_name": "EthioBBPE_AmharicBible",
  "use_checkpoint": true,
  "checkpoint_dir": "./models/checkpoints",
  "save_compressed": true,
  "compression_format": "gzip",
  "compression_level": 9,
  "checkpoint_steps": null,
  "num_threads": -1,
  "enable_backup": true,
  "max_checkpoints": 5,
  "enable_quantization": true,
  "quantization_bits": 8
}
```

## Model Files
- `tokenizer.json`: Standard tokenizer file (required)
- `vocab.json.gz`: Compressed vocabulary (optional, smaller size)
- `config.json`: Training configuration
- `training_metrics.json`: Training statistics
- `special_tokens_map.json`: Special tokens mapping
- `README.md`: This file

## Checkpoints
Checkpoints are saved in the `models/checkpoints` directory with metadata including:
- Checkpoint ID and timestamp
- Vocabulary size
- SHA256 checksum for integrity verification
- Training metrics at checkpoint time