File size: 1,856 Bytes
bb4c669 0cce5e4 bb4c669 0cce5e4 bb4c669 0cce5e4 bb4c669 0cce5e4 bb4c669 0cce5e4 bb4c669 0cce5e4 bb4c669 0cce5e4 bb4c669 0cce5e4 bb4c669 0cce5e4 bb4c669 0cce5e4 bb4c669 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | ---
language:
- code
license: mit
tags:
- byte-level-bpe
- tokenizer
- bbpe
- tokenizers
pipeline_tag: token-classification
library_name: tokenizers
datasets: []
metrics:
- vocabulary-size
---
# EthioBBPE: Byte-Level BPE Tokenizer
This is a Byte-Level BPE (BBPE) tokenizer trained using Hugging Face's `tokenizers` library. It handles diverse Unicode scripts and complex morphological structures seamlessly.
## Features
- **Byte-Level Encoding**: Robust against unknown characters, ensuring no `<UNK>` tokens
- **Universal Script Support**: Handles any Unicode character efficiently
- **Hugging Face Compatible**: Directly usable with `transformers` models
- **Efficient**: Fast encoding/decoding with optimized C++ backend
## Installation
```bash
pip install tokenizers
```
## Usage
### Load the Tokenizer
```python
from tokenizers import Tokenizer
# Load from Hugging Face Hub
tokenizer = Tokenizer.from_pretrained("Nexuss0781/Ethio-BBPE")
# Encode text
text = "Hello world! This is a test."
encoded = tokenizer.encode(text)
print(f"Token IDs: {encoded.ids}")
print(f"Tokens: {encoded.tokens}")
# Decode back
decoded = tokenizer.decode(encoded.ids)
print(f"Decoded: {decoded}")
```
### Using with Transformers
```python
from transformers import AutoTokenizer
# Load as a fast tokenizer
tokenizer = AutoTokenizer.from_pretrained("Nexuss0781/Ethio-BBPE", use_fast=True)
# Tokenize
inputs = tokenizer("The quick brown fox jumps over the lazy dog.")
print(inputs)
```
## Training Details
- **Model Type**: Byte-Level BPE
- **Vocabulary Size**: 30,000 tokens
- **Minimum Frequency**: 2
- **Special Tokens**: `[PAD]`, `[UNK]`, `[CLS]`, `[SEP]`, `[MASK]`
## Repository Structure
The full training codebase is available at:
- **GitHub**: [nexuss0781/Ethio_BBPE](https://github.com/nexuss0781/Ethio_BBPE)
## License
MIT License
|