Agntro/Hy3-295B-A21B-TQ2Q

Tencent Hunyuan-3 (Hy3) — TQ2Q (4-level quaternary 2-bit MoE experts)

Activation-aware 4-level (quaternary) 2-bit quantization of the routed expert FFNs of tencent/Hy3 (295B total / 21B active MoE; 192 experts, top-8), packed into llama.cpp's TQ2_0 container. 83.3 GB. The layer-0 dense FFN, shared experts, attention, router, embeddings and the head stay at higher precision.

⚠️ Work in progress — read before use

• First eval is in — an honest mixed result (see Quality & speed below). Paired vs the community Q2_K GGUF: we win prefill (≈2×) and disk size (−24%) but lose decode and perplexity. Still an early release. We tested Q5_K/Q6_K non-expert variants to recover decode — they do not flip it (best +9% decode at Q5_K, still trailing the community Q2_K on decode and PPL), so we kept Q8_0 non-experts; MTP speculative decoding is the remaining decode lever (future work).
• Runtime: needs Hunyuan-3 (hyv3) architecture support — not in mainline llama.cpp yet, and not on our fork's default branch. Build from the specific branch: git clone -b hy_v3-support https://github.com/AgntroAI/llama.cpp (arch commit 0d4b01d). A plain clone of AgntroAI/llama.cpp (master) will not load this model. The GGUF arch id is hyv3 (no underscore); hy_v3 / HYV3ForCausalLM refers only to the HF model class.
• GPU support. Mainline llama.cpp has no merged TQ2_0 CUDA kernel, so there the 2-bit experts run on CPU (full -ngl 99 offload still gives a ≈4× hybrid decode speedup). We built a TQ2_0 CUDA kernel and MERGED it into our fork's master (AgntroAI/llama.cpp): experts in VRAM, ≈25× prefill (measured on the 35B), q=3 verified correct (GPU-vs-CPU KLD 0.0078), determinism confirmed, test-backend-ops all-green (44/44 tq2_0 cases). Giant GPU benchmarks are in progress (Hy3 is being measured GPU-native now). Metal (Apple) and AMD/ROCm kernels are in progress. See https://agntro.dev/posts/tq2q-on-cuda.html

The format in one paragraph

TQ2_0 is a 2-bit container: each weight is one of 4 code words, dequantized as d·(q−1). Pure ternary uses only 3 and wastes q=3. We use it (q=3 → +2d), so each 256-weight block carries {−α, 0, +α, ±2α} — a strict superset of ternary at identical bytes and kernel speed. TQ2Q is a human label, not a new quant type — the file is a valid TQ2_0 GGUF.

⚠️ Critical hazard — never requantize or merge into this GGUF

llama.cpp's quantize_row_tq2_0 clamps to [−1, +1], silently stripping the 4th level back to ternary — no error, quality gone. Run it for inference; never llama-quantize it or merge a LoRA and re-save. Need a different quant? Re-run from the source model.

Run it

Needs a hy_v3-capable llama.cpp (see the WIP note), CPU-only:

llama-cli -m Hy3-TQ2Q.gguf -ngl 0 -t <ncpu> --temp 0 --jinja -p "..."

≈83 GB of weights → ≈90 GB+ RAM.

Scope & limitations

• Only the routed expert FFNs are 2-bit (237 tensors → TQ2_0, q=3 present), re-fit with GPTQ (4-level, salience-weighted per-expert Hessians). Everything else is higher precision.
• Paired eval vs community Q2_K (vcruz305/Hy3-GGUF, ≈109 GB), same llama.cpp code path, CPU (-ngl 0): prefill pp512 114.6 vs 56.2 t/s (≈2× ours); decode tg128 11.7 vs 15.9 t/s (community faster); WikiText-2 PPL 6.39 vs 5.92 (community lower); MMLU/GSM8K tie. Net: we win prefill + disk (−24%), lose decode + perplexity. Cause: our non-expert tensors are Q8_0, which dominate single-token decode (only 8/192 experts fire per token). We tested Q5_K/Q6_K non-expert variants to recover decode — they do not flip it (best +9% decode at Q5_K, still trailing the community Q2_K on decode and PPL), so we kept Q8_0 non-experts; MTP speculative decoding is the remaining decode lever (future work).
• CUDA experts now run GPU-native on our fork's master (see the GPU-support note above); Metal (Apple) and AMD/ROCm TQ2_0 kernels are in progress, and on Vulkan the experts still run on CPU.

---

Base model tencent/Hy3 (Apache-2.0); this quantization inherits Apache-2.0. Code: https://github.com/AgntroAI/tq2-quaternary