vinpix/Bonsai-8B-TQ-GGUF

Bonsai-8B ternary GGUF (stock llama.cpp)

GGUF repacks of Ternary-Bonsai-8B that run on stock llama.cpp, LM Studio and Ollama with no fork required.

Original model: prism-ml/Ternary-Bonsai-8B-gguf Quantized with llama.cpp from the F16 source.

Two variants

File	Body	Bit-packing	Size
Bonsai-8B-TQ1_0-Q4out-llamacpp.gguf	TQ1_0	1.69 bpw	~2.02 GiB
Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf	TQ2_0	2.06 bpw	~2.32 GiB

Both use Q4_0 for output and token embeddings, and have RoPE/YaRN scaling removed (native context up to 16k).

Quality: the two variants are equivalent

TQ1_0 and TQ2_0 encode the same ternary weights {-1, 0, +1}; they differ only in how those weights are bit-packed. Both are lossless packings of identical values, so they are not just close — they are interchangeable in output.

Measured with KL-divergence (llama.cpp --kl-divergence) of TQ1_0 against TQ2_0 on wikitext-2:

• Same top-1 token: 100.0% — the two files pick the same most-likely token at every position tested
• Mean token-probability difference: ~0.000%
• Maximum token-probability difference: 0.005%

Perplexity on wikitext-2 is identical for both. The choice between them is purely size vs. compatibility, never quality.

Which to pick

• TQ1_0 — the smaller file (~300 MB less). Use it when disk/RAM footprint is the priority.
• TQ2_0 — the most broadly compatible option (the TQ2_0 tensor type has been in mainline llama.cpp longer, and its simpler byte-aligned packing is handled well across builds and backends).

If a TQ1_0 build ever produces garbled output on a particular setup (some backends have had issues with low-bit types), use the TQ2_0 file instead — the output is identical by construction.

Why this repack

The official PrismML Q2_0 release uses a g128 tensor type that only loads in the PrismML fork; on a stock llama.cpp build it fails with an unknown ggml type. These repacks use only mainline tensor types, so they load on any current llama.cpp, LM Studio or Ollama, today. (A similar compatibility repack is lilyanatia/Bonsai-8B-requantized.)

Quantization layout

The body is kept in its native ternary form. TQ1_0 and TQ2_0 both represent {-1, 0, +1} weights exactly (lossless), which is why their outputs match. Output and token-embedding tensors carry the only non-ternary information and are quantized to Q4_0 from the F16 source.

On this ternary model, Q4_0 gave the best perplexity-to-size trade-off for the output and embedding tensors. Because the underlying model is trained natively ternary, "richer" formats (Q5_K, Q6_K, IQ4_NL, and k-quants) measured worse perplexity than Q4_0 on these tensors, not better — higher-bit grids reintroduce quantization noise around weights that are already exact. Promoting any subset of the ternary body to higher precision likewise increases perplexity. The uniform native-ternary layout is the optimum here.

Run

# TQ1_0 (smaller)
llama-cli -m Bonsai-8B-TQ1_0-Q4out-llamacpp.gguf --temp 0.5 --top-p 0.85 --top-k 20 -p "Your prompt" -n 256

# TQ2_0 (widest compatibility)
llama-cli -m Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf --temp 0.5 --top-p 0.85 --top-k 20 -p "Your prompt" -n 256

For CPU, set -t to your physical core count. Recommended sampling defaults (temp 0.5, top_p 0.85, top_k 20) are baked into both files. On CPU, plain attention (no --flash-attn) is generally fine for generation.

Notes

• These are requantizations, not retrains; capabilities are inherited from Ternary-Bonsai-8B.
• The official PrismML g128 build packs the ternary weights at a finer group size (128 vs the 256 used by the mainline TQ types), which is marginally more precise on the weights but requires their fork; these repacks trade that for portability on stock tooling.
• For context above 16k, use the official build.