Hy3-GGUF

imatrix GGUF quantizations of tencent/Hy3 — 295B total / 21B active MoE (192 experts, top-8), 80 layers + 1 MTP/NextN layer (3.8B), 256K context.

Quantized day-zero from the BF16 release and smoke-tested on real hardware before upload (NVIDIA DGX Spark, GB10). Performance numbers below are measured, not estimated.

⚠️ Requires llama.cpp PR #25364 (unmerged)

The hy_v3 architecture is not yet in llama.cpp master. Until PR #25364 merges, build from the PR branch:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git fetch origin pull/25364/head:hy3-port && git checkout hy3-port
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j"$(nproc)"

These quants were produced at PR head a4da4b5cfdc4e5fa9def068e216a6e5154f22848.

Quants

All quants use an importance matrix (Hy3.imatrix, included) computed on a ~63KB diverse coding/reasoning/chat calibration corpus. Files are sharded at ~48GB for HF's 50GB limit — download the whole folder and point llama.cpp at the -00001-of-* shard; the rest load automatically.

Quant	Size	~BPW	Fits	Notes
Q8_0	318 GB	8.6	server-class	near-lossless
Q5_K_M	212 GB	5.8	2× 128GB-class
Q4_K_M	181 GB	4.9	2× 128GB-class	recommended dual-node; verified over RPC
IQ4_XS	159 GB	4.3	2× 128GB-class
Q3_K_M	143 GB	3.9	2× 128GB-class
IQ3_XXS	117 GB	3.2	single 128GB-class (borderline)	MTP block @ q8_0; best quality-per-GB single-box
Q2_K	109 GB	3.0	single 128GB-class
IQ2_M	100 GB	2.7	single 128GB-class	MTP block @ q8_0; smallest tier

On low-bit IQ tiers the MTP/NextN layer (blk.80.*) is kept at q8_0 (--tensor-type) — very-low-bit quantization of that block is not possible without imatrix coverage, and this preserves it intact for future speculative decoding support.

Measured performance (DGX Spark, GB10, 273GB/s unified memory)

Single node, fully GPU-resident, -fa on:

Quant	Gen tok/s	Prompt tok/s
IQ2_M	18.5	36.4
Q2_K	18.1	35.5
IQ3_XXS	17.1	32.4

Dual node (Q4_K_M, 181GB layer-split across 2× GB10 over 200GbE via llama.cpp RPC): 14.0 tok/s gen / 21.9 tok/s prompt. RPC layer-split adds capacity for bigger quants, not speed — expect single-node-or-slower decode rates.

All smoke-tested tiers (IQ2_M, Q2_K, IQ3_XXS single-node; Q4_K_M dual-node) produced coherent output (code generation + chat), with the Hy3 chat template engaging correctly via --jinja. The remaining tiers (Q8_0, Q5_K_M, IQ4_XS, Q3_K_M) were produced by the same verified pipeline but not individually load-tested — report any issues in the community tab.

Running

# chat/completion (--jinja is required — the Hy3 template is not natively supported)
./build/bin/llama-completion \
  -m Hy3-Q2_K/Hy3-Q2_K-00001-of-00003.gguf \
  -ngl 99 -fa on -c 8192 --jinja \
  -p "Write a Python function that returns the median of a list." -n 256

# server
./build/bin/llama-server \
  -m Hy3-Q2_K/Hy3-Q2_K-00001-of-00003.gguf \
  -ngl 99 -fa on -c 8192 --jinja --host 0.0.0.0 --port 8080

Tips:

• First load of a 100GB+ quant can take several minutes — don't kill it early.
• Constrained on memory? Offload MoE expert tensors to CPU with --n-cpu-moe N.
• --jinja matters: without it the chat template aborts on current llama.cpp.

MTP / speculative decoding status

Hy3's MTP (multi-token prediction) tensors are converted and stored in these GGUFs, but PR #25364 currently skips them in the forward graph — llama.cpp cannot yet use them for speculative decoding. The weights are preserved (q8_0 on low-bit tiers) so existing files become spec-decode-ready if/when the PR adds graph support. No re-download should be needed.

Provenance

• Source: tencent/Hy3 (BF16, 597.6GB, 99 shards)
• llama.cpp: PR #25364 @ a4da4b5cfdc4e5fa9def068e216a6e5154f22848
• imatrix computed on a Q8_0 intermediate (standard practice for models whose BF16 GGUF exceeds node RAM)
• Quantized and validated on a 2-node NVIDIA DGX Spark cluster

Quantized by vcruz305. Please report issues in the community tab.