TR-HASH Tokenizer 32K
The canonical tokenizer for the TR-HASH language-model line. It preserves the original 32,000-token ByteLevel BPE vocabulary exactly and appends four atomic reasoning-boundary tokens. No existing token was removed, renamed, or reindexed.
Vocabulary contract
| Component | Size / ID |
|---|---|
| Base ByteLevel BPE | 32,000 |
| `< | think_start |
| `< | think_end |
| `< | final_start |
| `< | final_end |
| Total tokenizer length | 32,004 |
The base BPE model, normalization, pre-tokenization, post-processing, decoder, and IDs 0–31,999 are byte-for-byte compatible with the original release.
Properties
- Model: ByteLevel BPE
- Normalization: Unicode NFC
- EOS:
</s>(ID 0) - PAD:
<pad>(ID 1) - BOS:
<s>(ID 2) - UNK:
<unk>(ID 3) - Declared model context: 2,048 tokens
- No chat template is imposed by this tokenizer repository
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"AETHORIA-AI/TR-HASH-Tokenizer-32K"
)
assert len(tokenizer) == 32004
assert tokenizer.convert_tokens_to_ids("<|think_start|>") == 32000
assert tokenizer.convert_tokens_to_ids("<|final_end|>") == 32003
Each canonical TR-HASH 200M model repository also carries a local copy of these tokenizer files so it remains loadable offline without depending on this repository at runtime.
Compatibility and migration
TR-HASH routes tokens deterministically from token IDs. Appending IDs
32,000–32,003 leaves every old token ID and every old route unchanged. Model
checkpoints still require their embedding matrix and persisted routing buffers
to be expanded together; replacing only tokenizer.json is not sufficient.
The four new model embedding rows are initialized from the mean of the same strings' old-tokenizer subtoken embeddings. They are structural initialization, not evidence that a checkpoint has already learned a reasoning protocol.
Comparison with GPT r50k_base
r50k_base is used here as the approximately 50K-token GPT reference. The
comparison covers the unchanged 32,000-token BPE model; the four appended
control tokens do not alter ordinary-text compression.
| Encoding | Base BPE vocabulary | Tokens on fixed suite | Characters/token |
|---|---|---|---|
| TR-HASH Tokenizer 32K | 32,000 | 375 | 3.712 |
GPT r50k_base |
50,257 | 354 | 3.932 |
On the fixed 1,392-character suite included in
benchmark_r50k.py, TR-HASH produces 5.9% more tokens
than r50k_base. This illustrative suite covers English prose, technical text,
code/JSON, mathematics, and French; it is not a universal compression claim.
pip install transformers tiktoken
python benchmark_r50k.py
Associated dataset
- Downloads last month
- 19
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("AETHORIA-AI/TR-HASH-Tokenizer-32K", device_map="auto")