nano-kyrgyz

Kyrgyz language models written from primitives in PyTorch and trained on a laptop.

Not a fine-tune. Not a distillation. RMSNorm, RoPE, grouped-query attention, QK-norm, SwiGLU, the Muon optimiser, the INT4 quantiser, the KV cache — every component is implemented from scratch in the source repository, and each is ablated rather than assumed.

Kyrgyz is a low-resource Turkic language: ~5M speakers, and three Cyrillic letters (ң ө ү) that mainstream OCR and tokenisation stacks systematically corrupt. Everything here was trained on an Apple laptop, for $0.

---

Three generations, one honest metric

All numbers are bits per character on held-out data, so models with different tokenizers are directly comparable — bpc = nats/token × (tokens/chars) ÷ ln 2.

A uniform-random model is not one number here, because "uniform over the vocabulary" means something different for a 151-symbol character vocabulary than for an 8,192-token BPE one: 7.24 bpc random for v0.1, 3.25 bpc for v0.2. Both models beat their own baseline by a wide margin; the bpc column is what compares them to each other.

	v0.1 base	v0.1 chat	v0.2 base	v0.2 chat
parameters	5.7M	5.7M	42.2M	42.2M
tokenizer	char, 151	char, 152	byte-level BPE, 8,192	byte-level BPE, 8,192
context	256 chars	256 chars	512 tokens ≈ 2,050 chars	512 tokens
training tokens	49M	+SFT	786.4M (5.34 epochs)	v4 base (2.0B-char corpus) + SFT
val bits/char	1.737	—	1.129	—
orthography (ң ө ү)	98.1%	—	93.3%	—
repeated 4-grams, greedy ↓	44.6%	—	27.4%	—
repeated 4-grams, temp 0.7 ↓	0.7%	—	4.4%	—
stops on its own	✗	✓	✓	✓

Orthography is the metric this project cares most about: real held-out sentences are re-spelled with the confusable letter (ө→о, ү→у, ң→н) and the model must assign higher likelihood to the true spelling. These are the exact letters that Russian-trained and OCR stacks collapse. Scored by length-normalised sequence log-likelihood so char-level and BPE models are compared the same way — though note a char model perturbs exactly one token where BPE re-tokenises the word, which flatters the char model somewhat, and is most of why v0.1 leads this row.

Per letter, v0.2: ө/о 97.0%, ү/у 94.5%, ң/н 89.8%. ң is consistently the hardest of the three — it is the rarest, it never begins a word, and its capital form appears about five times in ten million characters.

Repeated 4-grams measures degeneration (lower is better), over an equal ~400-character budget per model — a fixed token budget is not comparable across tokenizers and would give the BPE model four times as many windows in which to repeat. Greedy is reported because it is deterministic and comparable; temperature 0.7 is reported because it is what you would actually serve. Both are produced by scripts/eval_suite.py; nothing here is hand-entered.

At this size, greedy decoding loops on list-, title- and boilerplate-shaped prompts ("Бишкек шаары" invites an address; a heading invites its own repetition). Sampling largely — not entirely — fixes it: at --temp 0.7 --top-k 40 the degeneration rate falls from 27% to 3%. The residue is long institutional/legal boilerplate, where the model can still lock into a phrase:

Кыргыз Республикасынын Өкмөтү менен Сауд Арабиясы Өкмөтүнүн ортосундагы
Кыргыз Республикасынын Өкмөтүнүн ортосундагы Сауд Арабия Өкмөтүнүн …   ← loops
Кечээ Бишкекте өткөн жыйында “Коопсуздук кеңеши жөнүндө” мыйзам долбоору
талкууланды.                                                            ← stops cleanly

Both are real, unedited outputs at the settings above. More data and parameters are what fix the first case; this is a 42M-parameter model.

v0.2 was trained on a 588.8M-character corpus (Leipzig Wikipedia + NewsCrawl + CC-100 ky), 57× the v0.1 corpus, cleaned through NFC normalisation, homoglyph repair, a Cyrillic-ratio filter and global dedup — with per-source provenance counts printed by the builder script.

---

Files

file	what it is	size
nano-kyrgyz-v3-q8_0.gguf	v0.2 base — start here	45 MB
nano-kyrgyz-v3-Q4_K_M.gguf	v0.2 base, 4-bit — smallest useful	28 MB
nano-kyrgyz-v3-f32.gguf	v0.2 base, unquantised	169 MB
nano-kyrgyz-q8_0.gguf	v0.1 base (5.7M, char-level)	6.1 MB
nano-kyrgyz-v3-chat-q8_0.gguf	v0.2 chat — Kyrgyz conversation + document Q&A, start here for chat	45.4 MB
nano-kyrgyz-v3-chat-Q4_K_M.gguf	v0.2 chat, 4-bit	28.2 MB
nano-kyrgyz-v3-chat-f32.gguf	v0.2 chat, unquantised	169.4 MB
vision/vlm_ckpt.pt, vision/yolo_layout_best.pt, vision/glyph_ckpt.pt	document pipeline checkpoints (PyTorch; run from the source repo)	40 / 5.5 / 15 MB
nano-kyrgyz-chat-q8_0.gguf	v0.1 chat (5.7M, char-level)	6.1 MB
nano-kyrgyz-f32.gguf, nano-kyrgyz-chat-f32.gguf	v0.1 unquantised	22.9 MB each

For conversation use nano-kyrgyz-v3-chat-q8_0.gguf (v0.2 chat, 42M) — it answers in Kyrgyz, can read a document's field map you paste in, and says when it does not know. The v0.1 chat file is kept for reference. For raw text continuation use the v0.2 base and prompt it with the start of a sentence.

---

Usage

LM Studio

Search DJKG/nano-kyrgyz in Discover, or drop a GGUF into ~/.lmstudio/models/DJKG/nano-kyrgyz/.

Both models are self-configuring — no prompt-template setup. The chat model ships its jinja template; the base models ship a deliberately empty one, so a base model behaves as a pure continuator in the chat UI instead of being wrapped in a turn format it never saw.

Set context to the model's trained maximum (512 for v0.2, 256 for v0.1) and turn off "Chat with Files" — RAG injection overflows these windows instantly.

llama.cpp

# v0.2 base — give it the beginning of a sentence
llama-completion -m nano-kyrgyz-v3-q8_0.gguf \
  -p "Кыргыз Республикасынын" -n 200 --temp 0.7 --no-conversation

# v0.2 chat — needs --jinja for the embedded template (context 512)
llama-cli -m nano-kyrgyz-v3-chat-q8_0.gguf --jinja -c 512
# v0.1 chat (char-level, context 256)
llama-cli -m nano-kyrgyz-chat-q8_0.gguf --jinja -c 256

Homebrew's llama-cli is an interactive TUI and hangs in scripts. Use llama-completion for non-interactive runs; the flag is --no-conversation.

Ollama

Modelfiles live in the source repo (export/Modelfile*):

ollama create nano-kyrgyz-chat -f export/Modelfile.chat
ollama run nano-kyrgyz-chat "Салам!"

OpenAI-compatible API (LM Studio server)

curl http://localhost:1234/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"nano-kyrgyz-chat",
       "messages":[{"role":"user","content":"Салам!"}],
       "temperature":0}'
# → "Салам! Кандай жардам керек?"

Python

from huggingface_hub import hf_hub_download
path = hf_hub_download("DJKG/nano-kyrgyz", "nano-kyrgyz-v3-q8_0.gguf")

---

What the chat model (v0.1) can and cannot do

Its SFT mix is deliberately grounded — every answer is contained in, or derivable from, the prompt. Teaching a 5.7M model to answer knowledge questions teaches it to hallucinate confidently, which is the exact failure the wider project exists to prevent.

Prompt	Response
Салам!	Салам! Кандай жардам керек?
Сен кимсиң?	Мен нано-кыргыз — кичинекей кыргызча тил модели.
Баш тамга менен жаз: бишкек	БИШКЕК
Кайталап жаз: Кыргызстан тоолуу өлкө.	Кыргызстан тоолуу өлкө.

Not trained, will produce nonsense: factual questions, arithmetic, reasoning, translation, Russian or English.

---

v0.2 chat — Kyrgyz conversation and document Q&A

nano-kyrgyz-v3-chat-*.gguf is the 42M BPE base (v4, 1.062 bpc) instruction-tuned with scripts/chat_sft_v3.py: response-only loss, half of the data multi-turn, best-checkpoint selection, and — new — document question-answering and abstention. Held-out conversation loss 0.450 nats/token (base 2.837).

Its data is deliberately grounded, so it learns a format, not facts it does not have:

share	task	example
35%	grounded string tasks (echo, continue, first sentence, extract numbers)	Тексттеги сандарды жаз: …
35%	document Q&A over the field map the vision pipeline emits	Документ:\nСумма: 538,29 сом\n…\n\nСуроо: Сумма канча? → 538,29 сом
12%	case conversion (ң ө ү round-trip)	Баш тамга менен жаз: өмүрбеков → ӨМҮРБЕКОВ
18%	identity, greetings, abstention (facts, arithmetic, other languages)	Кыргызстандын калкы канча? → Кечиресиз, мен так сандарды билбейм…

Document Q&A pairs are generated from the same synthetic renderer that trains the OCR (data/synth/*.jsonl), so field names, value formats and letters match what the pipeline actually emits. Questions about a field the document does not have are answered Жок, бул документте … көрсөтүлгөн эмес. — a 42M model that says so beats one that invents an account number.

Serving contract (byte-level BPE): the generation prompt ends at Жооп: with no trailing space — the answer's first token carries the space (ĠСалам). The embedded Jinja template (| trim on history, so echoed answers do not double the space) and the Ollama Modelfile both render this way, and tests/test_correctness.py asserts the template is byte-identical to the SFT data. Serve the format that was trained.

The 20-prompt exit sheet (scripts/chat_sheet_v3.py, answers saved verbatim in assets/chat_v3_sheet.json): 16/20 acceptable by hand scoring — misses are 5 (exact echo garbled), 6 and 7 (case conversion: a BPE model sees бишкек/БИШКЕК as unrelated tokens), 8 (one digit slip in number extraction); greetings, identity, all 7 document Q&A items, all 3 abstentions and multi-turn are correct

llama-cli -m nano-kyrgyz-v3-chat-q8_0.gguf --jinja -c 512
# from the source repo, with a photo:
python -m vision.ask --image bill.jpg --question "Сумма канча?"   # photo → OCR → answer

Not trained, will produce nonsense: open-domain facts, arithmetic, translation, Russian/English (it will tell you so, in Kyrgyz), and anything about a document it was not shown in the prompt. Exact copying and case conversion are weaker than in the char-level v0.1 chat — a BPE model sees бишкек and БИШКЕК as unrelated tokens.

---

Document understanding — the vision side (v2)

The source repo ships a from-scratch document pipeline: YOLO11n layout detection (9 classes) → line crops → nano-VLM OCR (ViT + pixel shuffle + projector + the v0.1 decoder, ~10M params) → reading order → label↔value linking → JSON, and vision/ask.py, which hands that JSON to the v0.2 chat model so you can ask about a photographed Kyrgyz form in Kyrgyz. The checkpoints are in the vision/ folder of this repo (PyTorch — the custom VLM cannot ship as a GGUF --mmproj).

What changed in v2, measured rather than asserted:

• Real handwriting in the training data. The renderer composites the handwriting class and ~30% of letter-only field values from 80,213 real handwritten Kyrgyz letters (Kyrgyz MNIST, CC BY-NC 4.0, all 36 letters incl. ң ө ү). Ground truth is exact by construction (uppercase — the dataset labels letters, not case); train pages use its train split, val pages its test split.
• Ink degradations (stroke thinning, fade, low resolution), each behind a legibility guard, added because a real-ink experiment showed Ө read as О 4.0% of the time vs О→Ө 0.2% — the missed Ө had 30% less ink in the bar. Lose the bar and a Kyrgyz letter silently becomes a Russian one.
• Digit-weighted loss and field-value oversampling in the OCR trainer; digits are ~10% of the characters and had a 66% error rate.

Same new validation set, three checkpoints (2,683 line crops, 152 real ink):

	CER	exact	digit ER	field_value CER	real-ink CER
v1 OCR model (trained on the old data)	28.0%	54.4%	65.9%	53.7%	153.6%
v2 data, v1 recipe	23.4%	54.6%	65.2%	46.7%	73.8%
v2 data + v2 recipe	19.7%	57.3%	53.6%	38.6%	73.6%

Layout detector on the same val set: 0.995 / 0.897 mAP@50 / mAP@50-95 after fine-tuning on v2 data at 1024 px (at 640: 0.881; v1 weights on the same set: 0.988 / 0.857). Isolated real handwritten letters with the same ViT: 94.7% top-1 on 17,860 held out.

OCR confidence gate. The reader returns a per-line confidence (exp mean token log-prob); the pipeline withholds fields below the gate (default 0.8) from the chat model and flags them for review, and vision/ask.py answers "could not read this reliably — check: «…»" for them. Coverage-at-precision on the v2 val set: at ≥0.8 it keeps 81.8% of lines at 70.0% exact (unfiltered 57.3%); field values 59.6% kept at 25.4% exact (unfiltered 15.3%), 32.8% at 40.5% for ≥0.9. A user guide (nano-kyrgyz-guide-en.pdf / -ru.pdf in the source repo) walks through every way to use these files — LM Studio, llama.cpp, Ollama, the API, and the vision pipeline step by step.

Still synthetic-only. Everything above is measured on rendered pages (now containing real ink glyphs); there is no hand-transcribed real-photo set yet, and that remains the biggest gap in the vision track.

---

Shipped as architecture qwen3, and why that is honest

These GGUFs declare general.architecture = "qwen3". They share no weights with any Qwen model and are not fine-tunes of one.

A bespoke architecture string would require landing an upstream llama.cpp PR before anyone could run the model. Instead the model was built from the same components as Qwen3 — RMSNorm, NEOX-style RoPE, GQA, per-head QK-norm before RoPE, SwiGLU — so llama.cpp's existing Qwen3 graph executes these weights natively.

The bar for calling that legitimate is token parity, not "it runs": at --temp 0, llama.cpp produces character-identical output to the reference PyTorch generate(), ң included.

---

Three bugs worth reading about

The engineering value of this project is in what measurement caught.

1. A frequency cutoff silently deleted a letter of the alphabet. A count >= 200 vocabulary threshold dropped capital Ң, which occurs 5 times in 10 million characters — Turkic phonotactics: /ŋ/ never begins a word, so the capital appears only in all-caps headings, which is exactly what official Kyrgyz forms look like. Nothing would have crashed; every future Ң becomes <unk>.

2. The corpus was contaminated with homoglyphs. Sampling at high temperature produced кѳйчщер — ѳ (U+0473 FITA), an archaic Russian letter abolished in 1918, not ө (U+04E9). 482 of them, plus 3,156 Latin o and 4,141 Latin e inside Cyrillic words. Total repaired: 0.0102% of the corpus — which is why nobody finds it by reading samples.

3. add_bos_token was wrong for the base model. Pretraining is a raw token river containing no <bos>; serving one placed a never-seen embedding at position 0 — the position every later token attends to — and the model degenerated:

add_bos_token = true   →  "Кыргыз Республикасынын Кыргыз Республикасынын Кыргыз Респуб"
add_bos_token = false  →  "Кыргыз Республикасынын Кол жазмалар фондусунда калыптанган."

A repetition loop on the project's own headline example. It survived verification because the parity check passed — llama.cpp matched PyTorch while both were fed the same wrong input. Parity against a reference proves the tensor map, not the serving contract. The repo now asserts serving flags against the training data instead.

---

Limitations

Stated plainly, because a card that hides these is worth less than one that names them.

• These models hold format and orthography, not facts. Factual questions produce fluent, confident, wrong answers. Inherent to the scale.
• Kyrgyz only. Russian and English input produce garbage.
• The chat models follow a dialogue format; they are not general assistants. v0.2 chat (42M) answers grounded questions and abstains; it does not know facts.
• v0.1 base has no EOS — it was never trained to stop; cap output length. v0.2 stops correctly.
• News/web/encyclopedic register, inherited from the corpus.
• No instruction-tuning, no RLHF, no safety tuning on any of these.
• Corpus dedup is exact-match; near-duplicate (minhash) filtering is deliberately deferred and disclosed.

---

Training

Data	Leipzig Wikipedia 2021 + NewsCrawl 2011/2016 + CC-100 ky — 588.8M chars, 2.91M unique lines
Cleaning	NFC → homoglyph repair → ≥70% Cyrillic → ң/ө/ү marker required → global dedup
Tokenizer	byte-level BPE, 8,192, specials at fixed ids; vocab size chosen by measured fertility
Precision	bf16, gradient accumulation, EMA, WSD schedule, full-state resume
Optimiser	Muon (Newton–Schulz orthogonalisation) on hidden matrices + AdamW on embeddings/head/norms
Hardware	Apple M5 Pro, MPS

Measured in the source repo: Muon beat AdamW by −0.126 val loss at equal steps; QK-norm earned its cost; z-loss slightly hurt at this scale; MQA matched GQA at half the KV cache. Several cut against received wisdom — which is the argument for running the ablation instead of copying a config.

---

Licence and attribution

Model weights: Apache 2.0.

Training data comes from the Leipzig Corpora Collection (CC BY-NC 4.0) and CC-100. The corpus licence is non-commercial; the weights are released as research output. Evaluate your own use accordingly.

Architecture is component-compatible with Qwen3 by deliberate design; no Qwen weights, data, or code were used.

Built by Bakyt Djumabaev.