File size: 2,318 Bytes
e7a5974 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 | ---
language:
- multilingual
tags:
- ethiobbpe
- bpe
- tokenizer
- byte-level
license: apache-2.0
datasets:
- user-provided
---
# EthioBBPE Tokenizer
This is a production-ready Byte-Level BPE tokenizer with advanced features for deployment.
## Features
- **Byte-Level**: Handles any Unicode character without <UNK>.
- **Multi-format Compression**: Supports gzip, bz2, and lzma compression.
- **Checkpointing**: Built-in safety checkpoints with metadata tracking.
- **Quantization**: Optional 8-bit/4-bit quantization for efficient deployment.
- **Training Metrics**: Comprehensive metrics tracking and logging.
- **Automatic Backup**: Checkpoint rotation to manage disk space.
## Usage
### Transformers
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("EthioBBPE_AmharicBible")
```
### Tokenizers Library
```python
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")
```
### Loading Compressed Vocab
```python
import gzip
import json
# Load compressed vocabulary
with gzip.open("vocab.json.gz", 'rt', encoding='utf-8') as f:
vocab = json.load(f)
```
## Training Configuration
```json
{
"vocab_size": 16000,
"min_frequency": 2,
"show_progress": true,
"special_tokens": [
"<pad>",
"<unk>",
"<s>",
"</s>",
"<mask>"
],
"lowercase": false,
"dropout": null,
"data_dir": "./data",
"model_save_dir": "models",
"model_name": "EthioBBPE_AmharicBible",
"use_checkpoint": true,
"checkpoint_dir": "./models/checkpoints",
"save_compressed": true,
"compression_format": "gzip",
"compression_level": 9,
"checkpoint_steps": null,
"num_threads": -1,
"enable_backup": true,
"max_checkpoints": 5,
"enable_quantization": true,
"quantization_bits": 8
}
```
## Model Files
- `tokenizer.json`: Standard tokenizer file (required)
- `vocab.json.gz`: Compressed vocabulary (optional, smaller size)
- `config.json`: Training configuration
- `training_metrics.json`: Training statistics
- `special_tokens_map.json`: Special tokens mapping
- `README.md`: This file
## Checkpoints
Checkpoints are saved in the `models/checkpoints` directory with metadata including:
- Checkpoint ID and timestamp
- Vocabulary size
- SHA256 checksum for integrity verification
- Training metrics at checkpoint time
|