Qwen3-1.7B-Base-ReTok

Qwen3-1.7B-Base-ReTok is a tokenizer-replaced and continued-pretrained variant of Qwen/Qwen3-1.7B-Base. The original Qwen tokenizer was replaced with a custom Piece tokenizer, then the model was recovered with continued pretraining.

This is the final tie-preserving v18 checkpoint from the Summer project.

Code, recipes, full reproduction	https://github.com/Ismantic/Summer
Design notes and known pitfalls	docs/WHY.md
Tokenizer (C++, ships the 81,903 vocab)	https://github.com/Ismantic/PieceTokenizer
Downstream translation models	https://github.com/Ismantic/Interpreter

Hugging Face repo id: Ismantic/Qwen3-1.7B-Base-ReTok

Important Tokenizer Note

This repository contains the custom tokenizer assets:

• Summer-Tokenizer.pt — the 81,903-piece vocabulary
• Summer-Tokenizer.dict.txt — Chinese segmentation dictionary. Not optional.
• token_mapping.json — pad / bos / eos ids
• tokenizer.py — the loader you should use

The model architecture loads through Transformers as Qwen3, but the tokenizer is not a standard Qwen tokenizer and AutoTokenizer will not work. Use the bundled wrapper:

from tokenizer import PieceTokenizerWrapper
tok = PieceTokenizerWrapper(".")          # the directory holding these files
ids = tok.encode("机器翻译的基本任务是")

Keep Summer-Tokenizer.dict.txt next to Summer-Tokenizer.pt. Without it Chinese text tokenizes to different ids — not just slower. Round-trip decoding still returns the original string, so the breakage is silent; the model simply receives input it was never trained on. The loader raises rather than falling back.

Running with vLLM

vLLM loads the weights fine — config.json declares Qwen3ForCausalLM, so vLLM uses its own Qwen3 implementation and maps weights by state-dict key. It cannot use the tokenizer, so pass skip_tokenizer_init=True and feed token ids yourself:

from vllm import LLM, SamplingParams
from vllm.inputs import TokensPrompt
from tokenizer import PieceTokenizerWrapper

tok = PieceTokenizerWrapper(".")
llm = LLM(model=".", skip_tokenizer_init=True, dtype="bfloat16")

ids = tok.encode("机器翻译的基本任务是", add_special_tokens=False)
out = llm.generate([TokensPrompt(prompt_token_ids=ids)],
                   SamplingParams(temperature=0.0, max_tokens=64,
                                  stop_token_ids=[tok.eos_token_id]))
print(tok.decode(list(out[0].outputs[0].token_ids)))

See example_vllm.py. vllm serve does not work out of the box — the OpenAI-compatible server needs to turn text into tokens and cannot do so with this vocabulary; callers must send token ids.

Files

model.safetensors	weights, 310 tensors (tied — no lm_head.weight)
Summer-Tokenizer.pt	the 81,903-piece vocabulary, same file as in PieceTokenizer's save/
Summer-Tokenizer.dict.txt	Chinese segmentation dictionary — required
model.py checkpoint.py tokenizer.py	pure-torch inference code, so no transformers and no safetensors needed
example_load.py example_vllm.py	runnable examples

Training Summary

1. Replaced the original Qwen3-1.7B-Base tokenizer with an 81,903-token Piece tokenizer.
2. Initialized new embeddings by mapping each new piece through the original Qwen tokenizer and averaging old embeddings.
3. Phase 1: trained new embeddings on about 999M packed tokens while freezing the transformer.
4. Phase 2: annealed on about 200M packed tokens with LoRA q/v adapters, Aurora, and tied embedding/head preservation.

The full reproduction record (data mix, hyperparameters, timings) lives in the GitHub repo under docs/reports/.

Evaluation

All numbers come from the vLLM backend. Do not mix backends — measured on the same base model, lambada differs by 0.0223 between transformers and vLLM, and the direction is not even consistent across tasks.

WMT22, 1000 samples, 5-shot (sacrebleu / COMET wmt22-da):

Model	zh-en BLEU	zh-en COMET	en-zh BLEU	en-zh COMET
Qwen3-1.7B-Base	22.34	0.8122	38.34	0.8597
ReTok v18 Phase 1	20.26	0.7821	35.16	0.8276
ReTok v18 Phase 2 tie (this model)	20.46	0.7933	36.03	0.8444

WMT23, full set:

Model	zh-en BLEU	zh-en COMET	en-zh BLEU	en-zh COMET
ReTok v18 Phase 1	19.13	0.7767	38.83	0.8198
ReTok v18 Phase 2 tie (this model)	19.60	0.7834	40.99	0.8377

BLEU is quoted to two decimals on purpose. vLLM's greedy decoding is not reproducible: over 6 runs of the same checkpoint the BLEU range is 0.10–0.13, so a difference of that order is noise, not a result. COMET is two orders of magnitude more stable and is the more reliable of the two.

General benchmarks (lm-evaluation-harness):

Model	LAMBADA	PIQA	ARC-C	HellaSwag	CEVAL	GSM8K
Qwen3-1.7B-Base	0.6513	0.7731	0.5512	0.6705	0.6560	0.6710
ReTok v18 Phase 1	0.5674	0.7301	0.5137	0.6375	0.6263	0.0341
ReTok v18 Phase 2 tie (this model)	0.5768	0.7367	0.5145	0.6389	0.6204	0.0349

Metrics: acc for LAMBADA and CEVAL, acc_norm for PIQA / ARC-C / HellaSwag, exact_match,strict-match for GSM8K. Shots: 0 / 5 / 25 / 10 / 5 / 5.

GSM8K is a known, permanent loss. Replacing the vocabulary breaks Qwen3's numeric tokenization, and neither phase recovers it. This is the price of the new vocabulary, not a regression to chase.

Limitations

• This is a base model, not an instruction-tuned assistant.
• Generic Hugging Face hosted inference may not work until the custom Piece tokenizer is packaged as a standard AutoTokenizer implementation.
• Results remain below the original Qwen3-1.7B-Base on the WMT22 translation sample after tokenizer replacement.

License

The base model Qwen/Qwen3-1.7B-Base is released under Apache 2.0. This derivative checkpoint is prepared with the same license.