Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -14,7 +14,7 @@ datasets:
|
|
| 14 |
- Nexuss0781/synaxarium
|
| 15 |
- Nexuss0781/conon-biblical-am-en
|
| 16 |
metrics:
|
| 17 |
-
-
|
| 18 |
widget:
|
| 19 |
- text: "ሰላም ለኢዮብ ዘኢነበበ ከንቶ ።"
|
| 20 |
---
|
|
@@ -25,17 +25,24 @@ widget:
|
|
| 25 |
[](https://huggingface.co/Nexuss0781/Ethio-BBPE)
|
| 26 |
[](https://en.wikipedia.org/wiki/Amharic)
|
| 27 |
[](https://huggingface.co/docs/tokenizers/index)
|
|
|
|
| 28 |
|
| 29 |
-
A production-ready **Byte-level BPE tokenizer** specifically trained on **Amharic biblical and religious texts**, achieving **
|
| 30 |
|
| 31 |
## ✨ Features
|
| 32 |
|
| 33 |
-
- ✅ **
|
| 34 |
- ✅ **Specialized Vocabulary**: Trained on 61,769 lines of Amharic biblical texts (Synaxarium + Canon Bible)
|
| 35 |
- ✅ **Compressed Storage**: Gzip compression (level 9) reduces model size by **89.8%** (1.3MB → 136KB)
|
| 36 |
- ✅ **Production Ready**: Checkpointing, metrics tracking, and comprehensive error handling
|
| 37 |
- ✅ **Ge'ez Script Support**: Full support for Ethiopic characters, numerals, and liturgical punctuation marks
|
| 38 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
## 📊 Training Data
|
| 40 |
|
| 41 |
| Dataset | Source | Texts | Description |
|
|
@@ -61,10 +68,10 @@ A production-ready **Byte-level BPE tokenizer** specifically trained on **Amhari
|
|
| 61 |
|
| 62 |
| Metric | Result |
|
| 63 |
|--------|--------|
|
| 64 |
-
| **
|
| 65 |
-
| **Ge'ez Punctuation** | ✅
|
| 66 |
-
| **Synaxarium Text** | ✅
|
| 67 |
-
| **Biblical Text** | ✅
|
| 68 |
| **Compression Ratio** | **89.8%** (1.3MB → 136KB) |
|
| 69 |
| **Training Time** | ~17 seconds |
|
| 70 |
|
|
@@ -189,7 +196,7 @@ for i, enc in enumerate(encodings):
|
|
| 189 |
|
| 190 |
## 🧪 Testing & Validation
|
| 191 |
|
| 192 |
-
All test cases achieve **
|
| 193 |
|
| 194 |
```python
|
| 195 |
test_cases = [
|
|
@@ -202,7 +209,7 @@ for name, text in test_cases:
|
|
| 202 |
encoded = tokenizer.encode(text)
|
| 203 |
decoded = tokenizer.decode(encoded.ids)
|
| 204 |
assert text == decoded, f"{name} failed!"
|
| 205 |
-
print(f"✅ {name}:
|
| 206 |
```
|
| 207 |
|
| 208 |
## 📚 Datasets
|
|
@@ -247,6 +254,7 @@ Apache License 2.0 - See [LICENSE](LICENSE) for details.
|
|
| 247 |
|
| 248 |
- **GitHub**: [nexuss0781/Ethio_BBPE](https://github.com/nexuss0781/Ethio_BBPE)
|
| 249 |
- **Hugging Face**: [Nexuss0781/Ethio-BBPE](https://huggingface.co/Nexuss0781/Ethio-BBPE)
|
|
|
|
| 250 |
- **Issues**: Please open an issue on GitHub for bugs or feature requests
|
| 251 |
|
| 252 |
---
|
|
|
|
| 14 |
- Nexuss0781/synaxarium
|
| 15 |
- Nexuss0781/conon-biblical-am-en
|
| 16 |
metrics:
|
| 17 |
+
- reconstruction-accuracy
|
| 18 |
widget:
|
| 19 |
- text: "ሰላም ለኢዮብ ዘኢነበበ ከንቶ ።"
|
| 20 |
---
|
|
|
|
| 25 |
[](https://huggingface.co/Nexuss0781/Ethio-BBPE)
|
| 26 |
[](https://en.wikipedia.org/wiki/Amharic)
|
| 27 |
[](https://huggingface.co/docs/tokenizers/index)
|
| 28 |
+
[](https://github.com/nexuss0781/Ethio_BBPE)
|
| 29 |
|
| 30 |
+
A production-ready **Byte-level BPE tokenizer** specifically trained on **Amharic biblical and religious texts**, achieving **accurate reconstruction** of complex Ge'ez script, ancient punctuation, and liturgical content.
|
| 31 |
|
| 32 |
## ✨ Features
|
| 33 |
|
| 34 |
+
- ✅ **Accurate Reconstruction**: High accuracy on all test samples including ancient Ge'ez punctuation
|
| 35 |
- ✅ **Specialized Vocabulary**: Trained on 61,769 lines of Amharic biblical texts (Synaxarium + Canon Bible)
|
| 36 |
- ✅ **Compressed Storage**: Gzip compression (level 9) reduces model size by **89.8%** (1.3MB → 136KB)
|
| 37 |
- ✅ **Production Ready**: Checkpointing, metrics tracking, and comprehensive error handling
|
| 38 |
- ✅ **Ge'ez Script Support**: Full support for Ethiopic characters, numerals, and liturgical punctuation marks
|
| 39 |
|
| 40 |
+
## 🔗 Resources
|
| 41 |
+
|
| 42 |
+
- **Source Code**: [GitHub Repository](https://github.com/nexuss0781/Ethio_BBPE)
|
| 43 |
+
- **Issue Tracker**: [GitHub Issues](https://github.com/nexuss0781/Ethio_BBPE/issues)
|
| 44 |
+
- **PyPI Package**: [EthioBBPE on PyPI](https://pypi.org/project/EthioBBPE/)
|
| 45 |
+
|
| 46 |
## 📊 Training Data
|
| 47 |
|
| 48 |
| Dataset | Source | Texts | Description |
|
|
|
|
| 68 |
|
| 69 |
| Metric | Result |
|
| 70 |
|--------|--------|
|
| 71 |
+
| **Accurate Reconstruction** | ✅ High accuracy |
|
| 72 |
+
| **Ge'ez Punctuation** | ✅ Accurate (1 token for `፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠`) |
|
| 73 |
+
| **Synaxarium Text** | ✅ Accurate (66 tokens) |
|
| 74 |
+
| **Biblical Text** | ✅ Accurate (82 tokens) |
|
| 75 |
| **Compression Ratio** | **89.8%** (1.3MB → 136KB) |
|
| 76 |
| **Training Time** | ~17 seconds |
|
| 77 |
|
|
|
|
| 196 |
|
| 197 |
## 🧪 Testing & Validation
|
| 198 |
|
| 199 |
+
All test cases achieve **accurate reconstruction**:
|
| 200 |
|
| 201 |
```python
|
| 202 |
test_cases = [
|
|
|
|
| 209 |
encoded = tokenizer.encode(text)
|
| 210 |
decoded = tokenizer.decode(encoded.ids)
|
| 211 |
assert text == decoded, f"{name} failed!"
|
| 212 |
+
print(f"✅ {name}: Accurate ({len(encoded.ids)} tokens)")
|
| 213 |
```
|
| 214 |
|
| 215 |
## 📚 Datasets
|
|
|
|
| 254 |
|
| 255 |
- **GitHub**: [nexuss0781/Ethio_BBPE](https://github.com/nexuss0781/Ethio_BBPE)
|
| 256 |
- **Hugging Face**: [Nexuss0781/Ethio-BBPE](https://huggingface.co/Nexuss0781/Ethio-BBPE)
|
| 257 |
+
- **PyPI**: [EthioBBPE Package](https://pypi.org/project/EthioBBPE/)
|
| 258 |
- **Issues**: Please open an issue on GitHub for bugs or feature requests
|
| 259 |
|
| 260 |
---
|