Post
19
First GGUF quants of Tencent's Hy3 (299B MoE), built before official llama.cpp support exists.
Hy3 dropped ~30 hours ago with only MLX and MXFP4 quants, both datacenter-sized. So I converted it myself using a community llama.cpp fork that implements the hy_v3 architecture.
What's in the repo:
- IQ1_M (62GB, fits a 128GB MacBook), IQ2_M (90GB), Q2_K (101GB), all with 1M context baked in via YaRN
- IQ quants are importance-matrix: bootstrap style. The static Q2_K ran RAM-resident to compute the imatrix, then IQ1_M and IQ2_M were requantized from the archived f16 with it
- Fixed chat template (the stock one uses .format() calls llama.cpp's Jinja rejects)
- Build instructions for the fork, including the two gotchas that cost me three build attempts
Honesty section, because that is how these repos work: this is EXPERIMENTAL. Not needle-certified yet (1M is baked but unverified, certification ladder will be published either way). MTP layer exists in the checkpoint but no llama.cpp build can run hy_v3 MTP inference yet, so it is not included. Real gate outputs are on the card, misses and all, judge for yourself.
satgeze/Hy3-1M-GGUF
Full quant ladder (Q3 through Q8) is mirroring to ModelScope for bigger hardware.
Hy3 dropped ~30 hours ago with only MLX and MXFP4 quants, both datacenter-sized. So I converted it myself using a community llama.cpp fork that implements the hy_v3 architecture.
What's in the repo:
- IQ1_M (62GB, fits a 128GB MacBook), IQ2_M (90GB), Q2_K (101GB), all with 1M context baked in via YaRN
- IQ quants are importance-matrix: bootstrap style. The static Q2_K ran RAM-resident to compute the imatrix, then IQ1_M and IQ2_M were requantized from the archived f16 with it
- Fixed chat template (the stock one uses .format() calls llama.cpp's Jinja rejects)
- Build instructions for the fork, including the two gotchas that cost me three build attempts
Honesty section, because that is how these repos work: this is EXPERIMENTAL. Not needle-certified yet (1M is baked but unverified, certification ladder will be published either way). MTP layer exists in the checkpoint but no llama.cpp build can run hy_v3 MTP inference yet, so it is not included. Real gate outputs are on the card, misses and all, judge for yourself.
satgeze/Hy3-1M-GGUF
Full quant ladder (Q3 through Q8) is mirroring to ModelScope for bigger hardware.
