BlackwoodAI/blackwood-0.7b-bpe49152-tokenizer

Byte-Level BPE Tokenizer (vocab = 49,152)

A production-grade byte-level BPE tokenizer trained on a mixture of permissively-licensed English text, code (via FIM) and math, with 256 reserved special-token slots covering ChatML / chain-of-thought / tool-call / document-boundary primitives.

The companion pretokenized corpus that uses this exact tokenizer.json is available at BlackwoodAI/blackwood-pretrain-49152.

At a glance

Property	Value
Vocab size	49,152
Algorithm	Byte-level Byte-Pair Encoding
Pre-tokenizer	ByteLevel (add_prefix_space=False)
Decoder	ByteLevel
Post-processor	ByteLevel (trim_offsets=False)
Initial alphabet	Full ByteLevel alphabet (256 bytes)
Min merge frequency	2
Unknown token	<|unk|> (rarely fires — byte-level fallback covers all UTF-8)
Special tokens	256 atomic tokens (IDs 0–255)
Library	tokenizers==0.21.0

The 256 special-token block is large by design — the first ~26 IDs cover common roles (chat, CoT, tool-call, document/source boundaries), and the remaining 230 slots are reserved for downstream fine-tune specialisations so new roles can be aliased without retraining BPE merges.

Special-token catalog (IDs 0–255)

256 reserved slots, stable across releases.

Core (IDs 0–2)

ID	Token	Purpose
0	<|endoftext|>	BOS / EOS / document boundary
1	<|pad|>	Right-padding
2	<|unk|>	Unknown (byte-level fallback handles all UTF-8)

Chat / ChatML (IDs 3–7)

ID	Token	Role
3	<|im_start|>	Message start
4	<|im_end|>	Message end
5	<|system|>	System role
6	<|user|>	User role
7	<|assistant|>	Assistant role

Reasoning / CoT (IDs 8–11)

ID	Token	Purpose
8	<think>	Open scratchpad
9	</think>	Close scratchpad
10	<answer>	Open final answer
11	</answer>	Close final answer

Tool calling — generic (IDs 12–13)

ID	Token	Purpose
12	<|tool_call|>	Open tool invocation
13	<|tool_result|>	Open tool result

Document structure (IDs 14–15)

ID	Token	Purpose
14	<|doc_sep|>	Document boundary within a packed sequence
15	<|source_sep|>	Source / domain boundary within a packed sequence

Extended markers (IDs 16–25)

For long-horizon tool / planning workflows. Compose with the generic tool-call wrappers.

ID	Token	Purpose
16	<|fn_name|>	Function name
17	<|fn_args|>	Function arguments
18	<|fn_response|>	Function return value
19	<|tool_error|>	Tool error signal
20	<|plan_step|>	Multi-step planning marker
21	<|reflect|>	Self-correction signal
22	<|memory|>	Persistent-memory access
23	<|search|>	Search marker
24	<|file_op|>	Filesystem operation marker
25	<|terminate|>	Clean stop

Reserved (IDs 26–255)

<\|reserved_010\|> through <\|reserved_239\|> — 230 slots reserved for downstream fine-tune specialisations.

Measured fertility (tok/char, real-corpus samples)

Numbers reproduced from test_report.json:

Source	tok/char	tok/byte
smollm-cosmopedia	0.186	0.185
smollm-fineweb-edu	0.204	0.204
dclm-edu	0.230	0.230
finemath	0.303	0.302
proof-pile-2	0.297	0.297
dolma3-mix-150B	0.272	0.272
olmo-mix-1124	0.227	0.226
dolma3-dolmino (stack-edu FIM)	0.237	0.235

Domain (synthetic)	tok/char
English prose	0.224
Python code	0.335
Rust code	0.392
LaTeX math	0.399
JSON	0.336
Chinese	0.821
Arabic	0.944
Emoji-heavy	0.614

Notes:

• English prose at 0.22 is competitive with Llama-3-class tokenizers.
• Non-Latin scripts (CJK, Arabic) are not optimised — this is an English / code / math tokenizer.
• Any UTF-8 input is covered losslessly thanks to ByteLevel.

Training corpus

Trained on 15 GB of UTF-8 text streamed across 7 sources (see stats.json for exact byte / doc counts):

Source	Share
smollm-corpus (cosmopedia-v2 + fineweb-edu-dedup)	30 %
dclm-edu	20 %
dolma3-dolmino-100B	15 %
finemath	15 %
proof-pile-2 (arxiv + algebraic-stack + open-web-math)	10 %
olmo-mix-1124	5 %
dolma3-mix-150B	5 %

All sources hold permissive licenses (ODC-BY / CC-BY-4.0 / various permissive).

Round-trip + correctness tests

Released with a 6-test verification suite:

1. Vocab size == 49,152
2. All 256 special tokens at correct sequential IDs (0–255)
3. Special tokens encode as single atomic tokens
4. Round-trip decode(encode(x)) == x exact on synthetic samples
5. Round-trip exact on real corpus samples across every source
6. ChatML / CoT / tool-call template strings parse as expected token streams

All six pass; details in test_report.json.

Usage

Raw tokenizers library

from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download

tok_path = hf_hub_download(
    repo_id="BlackwoodAI/blackwood-0.7b-bpe49152-tokenizer",
    filename="tokenizer.json",
)
tok = Tokenizer.from_file(tok_path)
ids = tok.encode("<|im_start|><|user|>hi<|im_end|>").ids
print([tok.id_to_token(i) for i in ids])
# ['<|im_start|>', '<|user|>', 'hi', '<|im_end|>']

transformers

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained(
    "BlackwoodAI/blackwood-0.7b-bpe49152-tokenizer",
    use_fast=True,
)

ChatML inference prompt

prompt = (
    "<|im_start|><|system|>You are a helpful coding assistant.<|im_end|>\n"
    "<|im_start|><|user|>Write fizzbuzz in Python.<|im_end|>\n"
    "<|im_start|><|assistant|>"
)
input_ids = tok.encode(prompt).ids

CoT template

prompt = (
    "<|im_start|><|user|>What's 17 × 23?<|im_end|>\n"
    "<|im_start|><|assistant|>"
    "<think>17 × 20 = 340, 17 × 3 = 51, total = 391.</think>"
    "<answer>391</answer><|im_end|>"
)

Tool-call template

prompt = (
    "<|im_start|><|user|>What's the weather in NYC?<|im_end|>\n"
    "<|im_start|><|assistant|>"
    "<|tool_call|><|fn_name|>get_weather"
    "<|fn_args|>{\"city\":\"NYC\"}"
    "<|tool_result|><|fn_response|>{\"temp_f\":72,\"condition\":\"clear\"}"
    "<answer>It's 72°F and clear in NYC.</answer>"
    "<|terminate|><|im_end|>"
)

Notes on byte-level encoding

• add_prefix_space=False: encoding "hello" differs from " hello" by the first token. Be consistent at training and inference.
• No normaliser is applied. Casing, accents, control bytes and Unicode variants are all preserved. Layer NFKC normalisation before encoding if your downstream needs it.
• No subword regularisation (BPE-Dropout, SentencePiece unigram sampling) is applied at the tokenizer level. Add it downstream if your use case needs it.

License

Tokenizer artifact: Apache-2.0. Training corpora retain their upstream licenses (ODC-BY / CC-BY-4.0 / various).