---
license: apache-2.0
base_model: Qwen/Qwen3-1.7B-Base
library_name: transformers
pipeline_tag: text-generation
tags:
- qwen3
- retok
- tokenizer-replacement
- continued-pretraining
- bilingual
language:
- en
- zh
---

# Qwen3-1.7B-Base-ReTok

Qwen3-1.7B-Base-ReTok is a tokenizer-replaced and continued-pretrained variant
of `Qwen/Qwen3-1.7B-Base`. The original Qwen tokenizer was replaced with a
custom Piece tokenizer, then the model was recovered with continued pretraining.

This is the final tie-preserving v18 checkpoint from the
[**Summer**](https://github.com/Ismantic/Summer) project.

| | |
|---|---|
| Code, recipes, full reproduction | https://github.com/Ismantic/Summer |
| Design notes and known pitfalls | [`docs/WHY.md`](https://github.com/Ismantic/Summer/blob/main/docs/WHY.md) |
| Tokenizer (C++, ships the 81,903 vocab) | https://github.com/Ismantic/PieceTokenizer |
| Downstream translation models | https://github.com/Ismantic/Interpreter |

Hugging Face repo id: `Ismantic/Qwen3-1.7B-Base-ReTok`

## Important Tokenizer Note

This repository contains the custom tokenizer assets:

- `Summer-Tokenizer.pt` — the 81,903-piece vocabulary
- `Summer-Tokenizer.dict.txt` — **Chinese segmentation dictionary. Not optional.**
- `token_mapping.json` — pad / bos / eos ids
- `tokenizer.py` — the loader you should use

The model architecture loads through Transformers as Qwen3, but **the tokenizer
is not a standard Qwen tokenizer and `AutoTokenizer` will not work.** Use the
bundled wrapper:

```python
from tokenizer import PieceTokenizerWrapper
tok = PieceTokenizerWrapper(".")          # the directory holding these files
ids = tok.encode("机器翻译的基本任务是")
```

**Keep `Summer-Tokenizer.dict.txt` next to `Summer-Tokenizer.pt`.** Without it
Chinese text tokenizes to *different ids* — not just slower. Round-trip decoding
still returns the original string, so the breakage is silent; the model simply
receives input it was never trained on. The loader raises rather than falling back.

## Running with vLLM

vLLM loads the **weights** fine — `config.json` declares `Qwen3ForCausalLM`, so
vLLM uses its own Qwen3 implementation and maps weights by state-dict key. It
cannot use the **tokenizer**, so pass `skip_tokenizer_init=True` and feed token
ids yourself:

```python
from vllm import LLM, SamplingParams
from vllm.inputs import TokensPrompt
from tokenizer import PieceTokenizerWrapper

tok = PieceTokenizerWrapper(".")
llm = LLM(model=".", skip_tokenizer_init=True, dtype="bfloat16")

ids = tok.encode("机器翻译的基本任务是", add_special_tokens=False)
out = llm.generate([TokensPrompt(prompt_token_ids=ids)],
                   SamplingParams(temperature=0.0, max_tokens=64,
                                  stop_token_ids=[tok.eos_token_id]))
print(tok.decode(list(out[0].outputs[0].token_ids)))
```

See `example_vllm.py`. **`vllm serve` does not work out of the box** — the
OpenAI-compatible server needs to turn text into tokens and cannot do so with
this vocabulary; callers must send token ids.

## Files

| | |
|---|---|
| `model.safetensors` | weights, 310 tensors (tied — no `lm_head.weight`) |
| `Summer-Tokenizer.pt` | the 81,903-piece vocabulary, same file as in [PieceTokenizer](https://github.com/Ismantic/PieceTokenizer)'s `save/` |
| `Summer-Tokenizer.dict.txt` | Chinese segmentation dictionary — **required** |
| `model.py` `checkpoint.py` `tokenizer.py` | pure-torch inference code, so **no `transformers` and no `safetensors` needed** |
| `example_load.py` `example_vllm.py` | runnable examples |

## Training Summary

1. Replaced the original Qwen3-1.7B-Base tokenizer with an 81,903-token Piece
   tokenizer.
2. Initialized new embeddings by mapping each new piece through the original
   Qwen tokenizer and averaging old embeddings.
3. Phase 1: trained new embeddings on about 999M packed tokens while freezing
   the transformer.
4. Phase 2: annealed on about 200M packed tokens with LoRA q/v adapters, Aurora,
   and tied embedding/head preservation.

The full reproduction record (data mix, hyperparameters, timings) lives in
the GitHub repo under `docs/reports/`.

## Evaluation

All numbers come from the vLLM backend. **Do not mix backends** — measured on
the same base model, lambada differs by 0.0223 between transformers and vLLM,
and the direction is not even consistent across tasks.

WMT22, 1000 samples, 5-shot (sacrebleu / COMET wmt22-da):

| Model | zh-en BLEU | zh-en COMET | en-zh BLEU | en-zh COMET |
| --- | ---: | ---: | ---: | ---: |
| Qwen3-1.7B-Base | 22.34 | 0.8122 | 38.34 | 0.8597 |
| ReTok v18 Phase 1 | 20.26 | 0.7821 | 35.16 | 0.8276 |
| **ReTok v18 Phase 2 tie** (this model) | **20.46** | **0.7933** | **36.03** | **0.8444** |

WMT23, full set:

| Model | zh-en BLEU | zh-en COMET | en-zh BLEU | en-zh COMET |
| --- | ---: | ---: | ---: | ---: |
| ReTok v18 Phase 1 | 19.13 | 0.7767 | 38.83 | 0.8198 |
| **ReTok v18 Phase 2 tie** (this model) | **19.60** | **0.7834** | **40.99** | **0.8377** |

**BLEU is quoted to two decimals on purpose.** vLLM's greedy decoding is not
reproducible: over 6 runs of the same checkpoint the BLEU range is 0.10–0.13,
so a difference of that order is noise, not a result. COMET is two orders of
magnitude more stable and is the more reliable of the two.

General benchmarks (lm-evaluation-harness):

| Model | LAMBADA | PIQA | ARC-C | HellaSwag | CEVAL | GSM8K |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Qwen3-1.7B-Base | 0.6513 | 0.7731 | 0.5512 | 0.6705 | 0.6560 | 0.6710 |
| ReTok v18 Phase 1 | 0.5674 | 0.7301 | 0.5137 | 0.6375 | 0.6263 | 0.0341 |
| **ReTok v18 Phase 2 tie** (this model) | **0.5768** | **0.7367** | **0.5145** | **0.6389** | **0.6204** | **0.0349** |

Metrics: `acc` for LAMBADA and CEVAL, `acc_norm` for PIQA / ARC-C / HellaSwag,
`exact_match,strict-match` for GSM8K. Shots: 0 / 5 / 25 / 10 / 5 / 5.

**GSM8K is a known, permanent loss.** Replacing the vocabulary breaks Qwen3's
numeric tokenization, and neither phase recovers it. This is the price of the
new vocabulary, not a regression to chase.

## Limitations

- This is a base model, not an instruction-tuned assistant.
- Generic Hugging Face hosted inference may not work until the custom Piece
  tokenizer is packaged as a standard `AutoTokenizer` implementation.
- Results remain below the original Qwen3-1.7B-Base on the WMT22 translation
  sample after tokenizer replacement.

## License

The base model `Qwen/Qwen3-1.7B-Base` is released under Apache 2.0. This
derivative checkpoint is prepared with the same license.