README.md

metadata

language:
  - multilingual
tags:
  - ethiobbpe
  - bpe
  - tokenizer
  - byte-level
license: apache-2.0
datasets:
  - user-provided

EthioBBPE Tokenizer

This is a production-ready Byte-Level BPE tokenizer with advanced features for deployment.

Features

• Byte-Level: Handles any Unicode character without .
• Multi-format Compression: Supports gzip, bz2, and lzma compression.
• Checkpointing: Built-in safety checkpoints with metadata tracking.
• Quantization: Optional 8-bit/4-bit quantization for efficient deployment.
• Training Metrics: Comprehensive metrics tracking and logging.
• Automatic Backup: Checkpoint rotation to manage disk space.

Usage

Transformers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("EthioBBPE_AmharicBible")

Tokenizers Library

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")

Loading Compressed Vocab

import gzip
import json

# Load compressed vocabulary
with gzip.open("vocab.json.gz", 'rt', encoding='utf-8') as f:
    vocab = json.load(f)

Training Configuration

{
  "vocab_size": 16000,
  "min_frequency": 2,
  "show_progress": true,
  "special_tokens": [
    "<pad>",
    "<unk>",
    "<s>",
    "</s>",
    "<mask>"
  ],
  "lowercase": false,
  "dropout": null,
  "data_dir": "./data",
  "model_save_dir": "models",
  "model_name": "EthioBBPE_AmharicBible",
  "use_checkpoint": true,
  "checkpoint_dir": "./models/checkpoints",
  "save_compressed": true,
  "compression_format": "gzip",
  "compression_level": 9,
  "checkpoint_steps": null,
  "num_threads": -1,
  "enable_backup": true,
  "max_checkpoints": 5,
  "enable_quantization": true,
  "quantization_bits": 8
}

Model Files

• tokenizer.json: Standard tokenizer file (required)
• vocab.json.gz: Compressed vocabulary (optional, smaller size)
• config.json: Training configuration
• training_metrics.json: Training statistics
• special_tokens_map.json: Special tokens mapping
• README.md: This file

Checkpoints

Checkpoints are saved in the models/checkpoints directory with metadata including:

• Checkpoint ID and timestamp
• Vocabulary size
• SHA256 checksum for integrity verification
• Training metrics at checkpoint time