Upload README.md with huggingface_hub

README.md

@@ -13,12 +13,15 @@ datasets:
 
 # EthioBBPE Tokenizer
 
-This is a production-ready Byte-Level BPE tokenizer trained for robust text processing.
+This is a production-ready Byte-Level BPE tokenizer with advanced features for deployment.
 
 ## Features
 - **Byte-Level**: Handles any Unicode character without <UNK>.
-- **Compressed Storage**: Supports gzip compression for efficient deployment.
-- **Checkpointing**: Built-in safety checkpoints during training.
+- **Multi-format Compression**: Supports gzip, bz2, and lzma compression.
+- **Checkpointing**: Built-in safety checkpoints with metadata tracking.
+- **Quantization**: Optional 8-bit/4-bit quantization for efficient deployment.
+- **Training Metrics**: Comprehensive metrics tracking and logging.
+- **Automatic Backup**: Checkpoint rotation to manage disk space.
 
 ## Usage
 
@@ -26,7 +29,7 @@ This is a production-ready Byte-Level BPE tokenizer trained for robust text proc
 ```python
 from transformers import AutoTokenizer
 
-tokenizer = AutoTokenizer.from_pretrained("EthioBBPE")
+tokenizer = AutoTokenizer.from_pretrained("EthioBBPE_AmharicBible")
 ```
 
 ### Tokenizers Library
@@ -36,28 +39,59 @@ from tokenizers import Tokenizer
 tokenizer = Tokenizer.from_file("tokenizer.json")
 ```
 
+### Loading Compressed Vocab
+```python
+import gzip
+import json
+
+# Load compressed vocabulary
+with gzip.open("vocab.json.gz", 'rt', encoding='utf-8') as f:
+    vocab = json.load(f)
+```
+
 ## Training Configuration
 ```json
 {
-  "vocab_size": 32000,
+  "vocab_size": 16000,
   "min_frequency": 2,
   "show_progress": true,
   "special_tokens": [
-    "[PAD]",
-    "[UNK]",
-    "[CLS]",
-    "[SEP]",
-    "[MASK]"
+    "<pad>",
+    "<unk>",
+    "<s>",
+    "</s>",
+    "<mask>"
   ],
   "lowercase": false,
   "dropout": null,
   "data_dir": "./data",
   "model_save_dir": "models",
-  "model_name": "EthioBBPE",
+  "model_name": "EthioBBPE_AmharicBible",
   "use_checkpoint": true,
   "checkpoint_dir": "./models/checkpoints",
   "save_compressed": true,
+  "compression_format": "gzip",
+  "compression_level": 9,
   "checkpoint_steps": null,
-  "num_threads": -1
+  "num_threads": -1,
+  "enable_backup": true,
+  "max_checkpoints": 5,
+  "enable_quantization": true,
+  "quantization_bits": 8
 }
 ```
+
+## Model Files
+- `tokenizer.json`: Standard tokenizer file (required)
+- `vocab.json.gz`: Compressed vocabulary (optional, smaller size)
+- `config.json`: Training configuration
+- `training_metrics.json`: Training statistics
+- `special_tokens_map.json`: Special tokens mapping
+- `README.md`: This file
+
+## Checkpoints
+Checkpoints are saved in the `models/checkpoints` directory with metadata including:
+- Checkpoint ID and timestamp
+- Vocabulary size
+- SHA256 checksum for integrity verification
+- Training metrics at checkpoint time