rost-1b-instruct-v2-GGUF

llama.cpp quantisations of rostlabs/rost-1b-instruct-v2, converted from revision 53597eef9d83dd2433d26fb737bf18dc648a06d4. The weights in that revision are byte-identical to d833f89f, the revision every published benchmark number was measured from.

file	size	notes
rost-1b-instruct-v2-BF16.gguf	2.58 GiB	the precision the model was trained in
rost-1b-instruct-v2-Q8_0.gguf	1.37 GiB
rost-1b-instruct-v2-Q6_K.gguf	1.06 GiB
rost-1b-instruct-v2-Q4_K_M.gguf	791 MiB	smallest

You need a llama.cpp that knows this architecture

llama.cpp compiles architectures in rather than loading them dynamically, so stock builds will not read these files. A fork with the one commit applied is at rostlabs/llama.cpp, whose master is upstream master plus model: add rost.

git clone -b master https://github.com/rostlabs/llama.cpp && cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j
./build/bin/llama-server --model rost-1b-instruct-v2-BF16.gguf --ctx-size 4096

Ollama and LM Studio will not work. Both bundle their own llama.cpp, so neither reads these files until the architecture is merged upstream.

How faithful is the conversion

The conversion is not a repacking. It absorbs the checkpoint's transposed rotary convention into the sign of the q and k projections and turns a weightless QK norm into ordinary norm weights, so the GGUF is an equivalent model built differently — and equivalence is a claim worth checking rather than assuming.

Checked against the greedy generations recorded by the release's repetition protocol, six prompts, temperature 0:

• Five of six reproduce character-for-character, including a 1,269-character essay and a 493-character English answer.
• One diverges, after 768 identical characters, on the seasons prompt: the recorded output reads Iarna (septembrie-noiembrie) and the GGUF reads Iarna (septembrie-30 noiembrie).

That prompt is the model's known degenerate case — it is the one prompt v2 loops on under every decoding setting tested, documented on the base model's card. Deep inside a low-confidence repetitive region, an argmax can flip on floating-point accumulation-order differences between two implementations. So the honest statement is near-exact agreement, not bitwise equivalence: over roughly two thousand characters of other output there is no difference at all, and the single difference appears where the model is least certain.

If you need guaranteed parity with the published benchmark numbers, use the safetensors model rather than a GGUF.

Decoding

The source repository deliberately prescribes no sampling parameters — no tested configuration passed its preregistered gate of zero looping and zero token-cap endings across six repetition-stress prompts. These files inherit that: they carry the tokenizer, both stop tokens and the chat template, and no sampling defaults. Choose your own, and see the base model's card for what was measured.

Licence

CC-BY-NC-4.0, inherited from the base model's training data.