---
language:
  - vi
license: other
library_name: transformers
tags:
  - vietnamese
  - tokenizer
  - byte-level-bpe
  - server-assistant
  - tool-use
  - logs
  - code
model_type: tokenizer
---

# aevynt/JupiMind-Dense-J1R-Tokenizer-24k

## Tong quan

Day la tokenizer rieng cho dong **JupiMind Dense J1R**. Muc tieu cua no la phuc vu:

- tieng Viet co dau
- technical va server text
- command line
- logs
- JSON, YAML, config
- tool-call va reasoning tags

Tokenizer nay duoc train de thay the hoan toan baseline cu von nen tieng Viet kem.

## Thiet ke

- loai: byte-level BPE
- byte fallback: bat
- vocab size: 24576
- context target: 4096
- special chat/tool tokens:
  - `<|im_start|>`, `<|im_end|>`
  - `<think>`, `</think>`
  - `<tool_call>`, `</tool_call>`
  - `<tool_response>`, `</tool_response>`
  - `<tools>`, `</tools>`

## Corpus train tokenizer

JSONL inputs:
- `/home/trainer/jupimind/outputs/pretrain/j1r_t4x4_mix.jsonl`

Seed text inputs:
- `/home/trainer/jupimind/data/tokenizer/j1r_server_seed.txt`
- `/home/trainer/jupimind/data/tokenizer/j1r_server_seed.txt`

## Compression benchmark

Tren sample evaluation:

- candidate avg tokens/doc: 702.29
- baseline avg tokens/doc: 2331.64
- improvement: 3.32x it token hon baseline
- candidate avg chars/token: 3.83
- baseline avg chars/token: 1.15

## Dung voi JupiMind

Tokenizer nay **khong tuong thich** voi checkpoint cu dung `vocab_size=6400`.

Muon dung that can:

1. cap nhat model config sang `vocab_size=24576`
2. pretrain lai tu dau hoac continued pretrain tu checkpoint cung vocab

## Open Source Files

Repo nay bao gom:

- `tokenizer.json`
- `tokenizer_config.json`
- `special_tokens_map.json`
- `jupimind_tokenizer_report.json`
- `README.md`