Nexuss0781 commited on
Commit
cd24241
·
verified ·
1 Parent(s): b1e0113

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +17 -9
README.md CHANGED
@@ -14,7 +14,7 @@ datasets:
14
  - Nexuss0781/synaxarium
15
  - Nexuss0781/conon-biblical-am-en
16
  metrics:
17
- - perfect-reconstruction
18
  widget:
19
  - text: "ሰላም ለኢዮብ ዘኢነበበ ከንቶ ።"
20
  ---
@@ -25,17 +25,24 @@ widget:
25
  [![Hugging Face](https://img.shields.io/badge/🤗-Hugging_Face-yellow)](https://huggingface.co/Nexuss0781/Ethio-BBPE)
26
  [![Amharic](https://img.shields.io/badge/Language-Amharic-green.svg)](https://en.wikipedia.org/wiki/Amharic)
27
  [![Tokenizer Type](https://img.shields.io/badge/Type-Byte--level_BPE-orange.svg)](https://huggingface.co/docs/tokenizers/index)
 
28
 
29
- A production-ready **Byte-level BPE tokenizer** specifically trained on **Amharic biblical and religious texts**, achieving **perfect reconstruction** of complex Ge'ez script, ancient punctuation, and liturgical content.
30
 
31
  ## ✨ Features
32
 
33
- - ✅ **Perfect Reconstruction**: 100% accuracy on all test samples including ancient Ge'ez punctuation
34
  - ✅ **Specialized Vocabulary**: Trained on 61,769 lines of Amharic biblical texts (Synaxarium + Canon Bible)
35
  - ✅ **Compressed Storage**: Gzip compression (level 9) reduces model size by **89.8%** (1.3MB → 136KB)
36
  - ✅ **Production Ready**: Checkpointing, metrics tracking, and comprehensive error handling
37
  - ✅ **Ge'ez Script Support**: Full support for Ethiopic characters, numerals, and liturgical punctuation marks
38
 
 
 
 
 
 
 
39
  ## 📊 Training Data
40
 
41
  | Dataset | Source | Texts | Description |
@@ -61,10 +68,10 @@ A production-ready **Byte-level BPE tokenizer** specifically trained on **Amhari
61
 
62
  | Metric | Result |
63
  |--------|--------|
64
- | **Perfect Reconstruction** | ✅ **100%** |
65
- | **Ge'ez Punctuation** | ✅ Perfect (1 token for `፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠`) |
66
- | **Synaxarium Text** | ✅ Perfect (66 tokens) |
67
- | **Biblical Text** | ✅ Perfect (82 tokens) |
68
  | **Compression Ratio** | **89.8%** (1.3MB → 136KB) |
69
  | **Training Time** | ~17 seconds |
70
 
@@ -189,7 +196,7 @@ for i, enc in enumerate(encodings):
189
 
190
  ## 🧪 Testing & Validation
191
 
192
- All test cases achieve **perfect reconstruction**:
193
 
194
  ```python
195
  test_cases = [
@@ -202,7 +209,7 @@ for name, text in test_cases:
202
  encoded = tokenizer.encode(text)
203
  decoded = tokenizer.decode(encoded.ids)
204
  assert text == decoded, f"{name} failed!"
205
- print(f"✅ {name}: Perfect ({len(encoded.ids)} tokens)")
206
  ```
207
 
208
  ## 📚 Datasets
@@ -247,6 +254,7 @@ Apache License 2.0 - See [LICENSE](LICENSE) for details.
247
 
248
  - **GitHub**: [nexuss0781/Ethio_BBPE](https://github.com/nexuss0781/Ethio_BBPE)
249
  - **Hugging Face**: [Nexuss0781/Ethio-BBPE](https://huggingface.co/Nexuss0781/Ethio-BBPE)
 
250
  - **Issues**: Please open an issue on GitHub for bugs or feature requests
251
 
252
  ---
 
14
  - Nexuss0781/synaxarium
15
  - Nexuss0781/conon-biblical-am-en
16
  metrics:
17
+ - reconstruction-accuracy
18
  widget:
19
  - text: "ሰላም ለኢዮብ ዘኢነበበ ከንቶ ።"
20
  ---
 
25
  [![Hugging Face](https://img.shields.io/badge/🤗-Hugging_Face-yellow)](https://huggingface.co/Nexuss0781/Ethio-BBPE)
26
  [![Amharic](https://img.shields.io/badge/Language-Amharic-green.svg)](https://en.wikipedia.org/wiki/Amharic)
27
  [![Tokenizer Type](https://img.shields.io/badge/Type-Byte--level_BPE-orange.svg)](https://huggingface.co/docs/tokenizers/index)
28
+ [![GitHub](https://img.shields.io/badge/GitHub-repo-blue)](https://github.com/nexuss0781/Ethio_BBPE)
29
 
30
+ A production-ready **Byte-level BPE tokenizer** specifically trained on **Amharic biblical and religious texts**, achieving **accurate reconstruction** of complex Ge'ez script, ancient punctuation, and liturgical content.
31
 
32
  ## ✨ Features
33
 
34
+ - ✅ **Accurate Reconstruction**: High accuracy on all test samples including ancient Ge'ez punctuation
35
  - ✅ **Specialized Vocabulary**: Trained on 61,769 lines of Amharic biblical texts (Synaxarium + Canon Bible)
36
  - ✅ **Compressed Storage**: Gzip compression (level 9) reduces model size by **89.8%** (1.3MB → 136KB)
37
  - ✅ **Production Ready**: Checkpointing, metrics tracking, and comprehensive error handling
38
  - ✅ **Ge'ez Script Support**: Full support for Ethiopic characters, numerals, and liturgical punctuation marks
39
 
40
+ ## 🔗 Resources
41
+
42
+ - **Source Code**: [GitHub Repository](https://github.com/nexuss0781/Ethio_BBPE)
43
+ - **Issue Tracker**: [GitHub Issues](https://github.com/nexuss0781/Ethio_BBPE/issues)
44
+ - **PyPI Package**: [EthioBBPE on PyPI](https://pypi.org/project/EthioBBPE/)
45
+
46
  ## 📊 Training Data
47
 
48
  | Dataset | Source | Texts | Description |
 
68
 
69
  | Metric | Result |
70
  |--------|--------|
71
+ | **Accurate Reconstruction** | ✅ High accuracy |
72
+ | **Ge'ez Punctuation** | ✅ Accurate (1 token for `፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠፠`) |
73
+ | **Synaxarium Text** | ✅ Accurate (66 tokens) |
74
+ | **Biblical Text** | ✅ Accurate (82 tokens) |
75
  | **Compression Ratio** | **89.8%** (1.3MB → 136KB) |
76
  | **Training Time** | ~17 seconds |
77
 
 
196
 
197
  ## 🧪 Testing & Validation
198
 
199
+ All test cases achieve **accurate reconstruction**:
200
 
201
  ```python
202
  test_cases = [
 
209
  encoded = tokenizer.encode(text)
210
  decoded = tokenizer.decode(encoded.ids)
211
  assert text == decoded, f"{name} failed!"
212
+ print(f"✅ {name}: Accurate ({len(encoded.ids)} tokens)")
213
  ```
214
 
215
  ## 📚 Datasets
 
254
 
255
  - **GitHub**: [nexuss0781/Ethio_BBPE](https://github.com/nexuss0781/Ethio_BBPE)
256
  - **Hugging Face**: [Nexuss0781/Ethio-BBPE](https://huggingface.co/Nexuss0781/Ethio-BBPE)
257
+ - **PyPI**: [EthioBBPE Package](https://pypi.org/project/EthioBBPE/)
258
  - **Issues**: Please open an issue on GitHub for bugs or feature requests
259
 
260
  ---