by then hopefully I'll have MTP also baked in. Hopefully 🤞
If happens, I'll push MTP quants. But my resources are not VRAM rich so that's why it takes this long in building/testing stuff.
Sat Grewal PRO
satgeze
AI & ML interests
None yet
Recent Activity
repliedto their post about 1 hour ago
First GGUF quants of Tencent's Hy3 (299B MoE), built before official llama.cpp support exists.
Hy3 dropped ~30 hours ago with only MLX and MXFP4 quants, both datacenter-sized. So I converted it myself using a community llama.cpp fork that implements the hy_v3 architecture.
What's in the repo:
- IQ1_M (62GB, fits a 128GB MacBook), IQ2_M (90GB), Q2_K (101GB), all with 1M context baked in via YaRN
- IQ quants are importance-matrix: bootstrap style. The static Q2_K ran RAM-resident to compute the imatrix, then IQ1_M and IQ2_M were requantized from the archived f16 with it
- Fixed chat template (the stock one uses .format() calls llama.cpp's Jinja rejects)
- Build instructions for the fork, including the two gotchas that cost me three build attempts
Honesty section, because that is how these repos work: this is EXPERIMENTAL. Not needle-certified yet (1M is baked but unverified, certification ladder will be published either way). MTP layer exists in the checkpoint but no llama.cpp build can run hy_v3 MTP inference yet, so it is not included. Real gate outputs are on the card, misses and all, judge for yourself.
https://huggingface.co/satgeze/Hy3-1M-GGUF
Full quant ladder (Q3 through Q8) is mirroring to ModelScope for bigger hardware. posted an update about 4 hours ago
First GGUF quants of Tencent's Hy3 (299B MoE), built before official llama.cpp support exists.
Hy3 dropped ~30 hours ago with only MLX and MXFP4 quants, both datacenter-sized. So I converted it myself using a community llama.cpp fork that implements the hy_v3 architecture.
What's in the repo:
- IQ1_M (62GB, fits a 128GB MacBook), IQ2_M (90GB), Q2_K (101GB), all with 1M context baked in via YaRN
- IQ quants are importance-matrix: bootstrap style. The static Q2_K ran RAM-resident to compute the imatrix, then IQ1_M and IQ2_M were requantized from the archived f16 with it
- Fixed chat template (the stock one uses .format() calls llama.cpp's Jinja rejects)
- Build instructions for the fork, including the two gotchas that cost me three build attempts
Honesty section, because that is how these repos work: this is EXPERIMENTAL. Not needle-certified yet (1M is baked but unverified, certification ladder will be published either way). MTP layer exists in the checkpoint but no llama.cpp build can run hy_v3 MTP inference yet, so it is not included. Real gate outputs are on the card, misses and all, judge for yourself.
https://huggingface.co/satgeze/Hy3-1M-GGUF
Full quant ladder (Q3 through Q8) is mirroring to ModelScope for bigger hardware. updated a model about 4 hours ago
satgeze/Hy3-1M-GGUFOrganizations
None yet
replied to their post about 1 hour ago
posted an update about 4 hours ago
Post
19
First GGUF quants of Tencent's Hy3 (299B MoE), built before official llama.cpp support exists.
Hy3 dropped ~30 hours ago with only MLX and MXFP4 quants, both datacenter-sized. So I converted it myself using a community llama.cpp fork that implements the hy_v3 architecture.
What's in the repo:
- IQ1_M (62GB, fits a 128GB MacBook), IQ2_M (90GB), Q2_K (101GB), all with 1M context baked in via YaRN
- IQ quants are importance-matrix: bootstrap style. The static Q2_K ran RAM-resident to compute the imatrix, then IQ1_M and IQ2_M were requantized from the archived f16 with it
- Fixed chat template (the stock one uses .format() calls llama.cpp's Jinja rejects)
- Build instructions for the fork, including the two gotchas that cost me three build attempts
Honesty section, because that is how these repos work: this is EXPERIMENTAL. Not needle-certified yet (1M is baked but unverified, certification ladder will be published either way). MTP layer exists in the checkpoint but no llama.cpp build can run hy_v3 MTP inference yet, so it is not included. Real gate outputs are on the card, misses and all, judge for yourself.
satgeze/Hy3-1M-GGUF
Full quant ladder (Q3 through Q8) is mirroring to ModelScope for bigger hardware.
Hy3 dropped ~30 hours ago with only MLX and MXFP4 quants, both datacenter-sized. So I converted it myself using a community llama.cpp fork that implements the hy_v3 architecture.
What's in the repo:
- IQ1_M (62GB, fits a 128GB MacBook), IQ2_M (90GB), Q2_K (101GB), all with 1M context baked in via YaRN
- IQ quants are importance-matrix: bootstrap style. The static Q2_K ran RAM-resident to compute the imatrix, then IQ1_M and IQ2_M were requantized from the archived f16 with it
- Fixed chat template (the stock one uses .format() calls llama.cpp's Jinja rejects)
- Build instructions for the fork, including the two gotchas that cost me three build attempts
Honesty section, because that is how these repos work: this is EXPERIMENTAL. Not needle-certified yet (1M is baked but unverified, certification ladder will be published either way). MTP layer exists in the checkpoint but no llama.cpp build can run hy_v3 MTP inference yet, so it is not included. Real gate outputs are on the card, misses and all, judge for yourself.
satgeze/Hy3-1M-GGUF
Full quant ladder (Q3 through Q8) is mirroring to ModelScope for bigger hardware.
- 2 replies
replied to their post about 17 hours ago
Good push! But small correction, it is not single-needle. My harness plants 10 needles at 5-95% depths and the model has to return ALL 10 in one single response. The niah_test.py and raw transcripts are right there in the repo, you can check the results.jsonl. So it's 10 parallel retrievals held together, not recency + lookup.
But yes, you are right on the deeper part. My 10 needles are independent facts, nothing forces the model to combine fact A with fact B. So it tests retrieval, not composition. I kinda like this idea, so I am adding a hop tier to the harness: chained needles (needle 1 has the key that unlocks needle 2), and variable tracking (one value changes few times across the context, ask the final state). Will run it at 500K and 1M on the same models and publish the raw results either way, misses included, like everything else in these repos.
posted an update 2 days ago
Post
118
satgeze/Ornith-1.0-35B-1M-GGUF
Ornith-1.0 with a 1,048,576-token context window, tested instead of claimed 🦜
Ornith is Qwen3.5-family under the hood, so YaRN factor 4 extends it from 262K native to exactly 1M. I baked that into the GGUF metadata (no fine-tuning, weights bit-identical) so llama.cpp and Ollama apply it with zero flags, then ran full needle-in-a-haystack ladders on my own hardware:
- satgeze/Ornith-1.0-35B-1M-GGUF: 10/10 needles at every rung from 32K through 1M, replicated with fresh seeds (M3 Max 128GB, ~6.8h cold 1M prefill)
- satgeze/Ornith-1.0-9B-1M-GGUF: perfect through 524K, honest 7/10 at 1M under Q4 + q8_0 KV, failure band charted in the card
- satgeze/Ornith-1.0-397B-1M-GGUF: IQ1_M through Q4_K_M as split GGUFs, coherence-gated
Also in the repos:
- Vision: Ornith kept the Qwen3.5 multimodal skeleton, so the VL vision tower (extracted by bartowski) attaches at runtime via llama-server --mmproj. OCR-tested on the 9B and 35B, mmproj files bundled.
- A measured residency matrix: on a single RTX 5090, every 9B quant up to Q6_K holds the full 1M window at 100 percent GPU, 162 to 244 tok/s.
- Quality gates: every low-bit quant passed a coherence test before upload. The 35B IQ1_S failed and was deleted rather than shipped.
Harness, method writeup, and raw per-needle data: https://github.com/satindergrewal/ornith-1m
All MIT. Credit to DeepReinforce for the models and bartowski for the imatrix quants and vision towers. If a config breaks retrieval for you, tell me and it goes in the card.
Ornith-1.0 with a 1,048,576-token context window, tested instead of claimed 🦜
Ornith is Qwen3.5-family under the hood, so YaRN factor 4 extends it from 262K native to exactly 1M. I baked that into the GGUF metadata (no fine-tuning, weights bit-identical) so llama.cpp and Ollama apply it with zero flags, then ran full needle-in-a-haystack ladders on my own hardware:
- satgeze/Ornith-1.0-35B-1M-GGUF: 10/10 needles at every rung from 32K through 1M, replicated with fresh seeds (M3 Max 128GB, ~6.8h cold 1M prefill)
- satgeze/Ornith-1.0-9B-1M-GGUF: perfect through 524K, honest 7/10 at 1M under Q4 + q8_0 KV, failure band charted in the card
- satgeze/Ornith-1.0-397B-1M-GGUF: IQ1_M through Q4_K_M as split GGUFs, coherence-gated
Also in the repos:
- Vision: Ornith kept the Qwen3.5 multimodal skeleton, so the VL vision tower (extracted by bartowski) attaches at runtime via llama-server --mmproj. OCR-tested on the 9B and 35B, mmproj files bundled.
- A measured residency matrix: on a single RTX 5090, every 9B quant up to Q6_K holds the full 1M window at 100 percent GPU, 162 to 244 tok/s.
- Quality gates: every low-bit quant passed a coherence test before upload. The 35B IQ1_S failed and was deleted rather than shipped.
Harness, method writeup, and raw per-needle data: https://github.com/satindergrewal/ornith-1m
All MIT. Credit to DeepReinforce for the models and bartowski for the imatrix quants and vision towers. If a config breaks retrieval for you, tell me and it goes in the card.
- 2 replies