---
license: apache-2.0
base_model: tencent/Hy3
base_model_relation: quantized
pipeline_tag: text-generation
tags:
  - gguf
  - llama.cpp
  - moe
  - imatrix
language:
  - en
  - zh
  - fr
---

# Hy3 295B-A21B — GGUF (IQ3_XXS-UD, imatrix)

GGUF quantization of [tencent/Hy3](https://huggingface.co/tencent/Hy3) (295B total / 21B active MoE, Apache 2.0), converted from the official FP8 checkpoint ([tencent/Hy3-FP8](https://huggingface.co/tencent/Hy3-FP8)).

> **Requires llama.cpp with `hy-v3` architecture support.** This is a brand-new architecture; support is not yet in a release. Until it is merged upstream, build llama.cpp from a branch that implements `HYV3ForCausalLM`. Once merged, any recent build works. <!-- TODO: link the PR once reopened -->

## Files

| file | size | recipe |
|---|---|---|
| `Hy3-UD128-*.gguf` | 116.7 GB total (sharded <45 GB) | routed experts: **down** IQ3_S · **gate** IQ2_S · **up** IQ3_XXS — attention Q5_K · shared expert + dense FFN Q6_K · output Q6_K · embeddings Q4_K |
| `Hy3-IQ4-UD-split-*.gguf` | 143.9 GB total (sharded <45 GB) | max-quality edition for dual-device rigs: routed down IQ4_XS · gate/up IQ3_S · attention/actives Q8_0 |
| `Hy3.imatrix.gguf` | small | importance matrix, ~125 chunks of 512 tokens, general-purpose calibration corpus |

The UD128 mix is an "unsloth-dynamic"-style asymmetric allocation. Tensors active on **every** token (attention, shared expert, dense layer-0 FFN, output head) keep high precision. Within the 192 routed experts — only 8 active per token, so they carry the low-bit budget — the extra bits go to the **down-projection** (`ffn_down_exps` at IQ3_S), the most quantization-sensitive expert tensor, funded by the **gate** projection at IQ2_S (a multiplicative mask, the most robust one); **up** sits in between at IQ3_XXS. Since routed experts are 97 % of the parameters, this is where the whole quality/size trade-off is decided.

Measured (wikitext-2 test, c=2048, 60 chunks, held-out from calibration): **PPL 4.797 ± 0.048** for this asymmetric mix vs 4.833 ± 0.048 for a flat IQ3_XXS-experts build at the same file size (Q8_0 reference ≈ 4.3) — the redistribution is measured, not assumed.

The imatrix file is published so you can requantize any other mix directly from a Q8_0 master without redoing the (expensive) calibration pass.

## Why this size

116.7 GB on disk, but the inert NextN layer (`blk.80`, ~2 GB) is never allocated at inference, so the resident footprint is **~107 GiB** — it fits **fully** in the GPU/unified memory of 128 GB-class machines (Strix Halo / Apple Silicon / multi-GPU rigs), no CPU offload needed. Usable unified RAM on "128 GB" machines is really ~122–126 GB (firmware/OS reserve varies), and whatever else runs on the box counts against you. Comfortable defaults: **24K–32K with q8_0 KV**, or **40K+ with q4_0 KV** — on recent llama.cpp builds the Hadamard-rotated KV cache makes q4_0 nearly q8_0-quality, at half the size (~87 KB/token vs ~160). For longer context, offload a few expert layers to CPU/second GPU.

## Running

Tencent's recommended sampling: `--temp 0.9 --top-p 1.0`. The chat template is embedded and resolved, so plain `--jinja` works.

Single device / unified memory:

```bash
llama-server -m Hy3-UD128-00001-of-00003.gguf -ngl 99 -fa on --jinja \
  -c 24576 -ctk q8_0 -ctv q8_0 --temp 0.9 --top-p 1.0   # or -c 40960 -ctk q4_0 -ctv q4_0
```

**AMD Strix Halo / RDNA3.5 iGPU tip:** run this model on the **Vulkan (RADV)** backend, not ROCm. Measured on a 128 GB Strix Halo (Radeon 8060S, llama-bench, same build, q4_0 KV): Vulkan 168 t/s prefill / 18.4 t/s decode vs ROCm 105 / 12.4 — and at 8K depth the ROCm flash-attention kernel collapses (26 t/s prefill vs 104 on Vulkan). ~17 t/s sustained decode in real serving at 50K context.

Split (small fast GPU + large slow memory — attention/dense/output on the fast GPU, routed experts on the big one), example CUDA + iGPU:

```bash
llama-server -m Hy3-IQ4-UD-split-00001-of-00004.gguf --device CUDA0,ROCm0 -ngl 99 -fa on --jinja \
  -c 16384 -ctk q4_0 -ctv q4_0 \
  -ot "ffn_.*_exps=ROCm0" -ot "output=CUDA0" -ot "token_embd=ROCm0" \
  --temp 0.9 --top-p 1.0
```

(Measured on Strix Halo iGPU + RTX 3090: 12.8 tok/s decode on the IQ4 split edition.)

## Notes

- The NextN/MTP layer (`blk.80`) is included but skipped at inference today, which keeps these files **MTP-ready**: community work on `hy_v3` MTP inference (llama.cpp PR #25395, ~97% draft acceptance reported) can use the draft head carried in these quants as-is — no re-download needed if/when it merges.
- **Requantizing this yourself?** The NextN layer (`blk.80`) has no imatrix data (it is never activated), so any very-low-bit type that requires an importance matrix — `iq3_xxs`, `iq2_*`, `iq1_*` — will abort on `blk.80.ffn_down_exps` at the end of the run. Pin that layer to an imatrix-exempt type first: `--tensor-type 'blk\.80\.=q4_0'` (its precision is irrelevant since it is skipped at inference). k-quants, `iq3_s`, and `iq4_xs` are unaffected.
- Quantized with a general-purpose calibration corpus — no domain-specific tuning.

## Provenance / verification

- Logit parity vs `transformers` verified at f32 (KL = 0.0, exact top-1 match on English/code/French/chat prompts); tokenizer verified against `AutoTokenizer`.
- Conversion chain: FP8 → Q8_0 master (317.6 GB) → imatrix → per-tensor requants.
- Recipe A/B'd on held-out wikitext-2 before release (see PPL above); IQ4 split edition sanity-checked 4/4 on adversarial math/code prompts.