early bench results
gave the Q3 a shot on one of our lab servers, appears it's quite prone to looping and also fails tool calls frequently. maybe this model is somewhat sensitive to quantization, reminds me a bit of minimax-m2.7.
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 190661 MiB):
Device 0: NVIDIA H100 NVL, compute capability 9.0, VMM: yes, VRAM: 95330 MiB
Device 1: NVIDIA H100 NVL, compute capability 9.0, VMM: yes, VRAM: 95330 MiB
| model | size | params | backend | ngl | fa | test | t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --------------: | -------------------: |
| hy-v3 295B.A21B Q3_K - Medium | 132.93 GiB | 298.79 B | CUDA | -1 | 1 | pp1024 | 964.78 ± 2.30 |
| hy-v3 295B.A21B Q3_K - Medium | 132.93 GiB | 298.79 B | CUDA | -1 | 1 | pp2048 | 1031.82 ± 1.04 |
| hy-v3 295B.A21B Q3_K - Medium | 132.93 GiB | 298.79 B | CUDA | -1 | 1 | tg128 | 58.02 ± 0.13 |
| hy-v3 295B.A21B Q3_K - Medium | 132.93 GiB | 298.79 B | CUDA | -1 | 1 | tg256 | 58.09 ± 0.03 |
| hy-v3 295B.A21B Q3_K - Medium | 132.93 GiB | 298.79 B | CUDA | -1 | 1 | tg512 | 57.94 ± 0.01 |
build: a4da4b5cf (9889)
Thanks for the H100 numbers — 58 tg / ~1000 pp on 2× NVL is a great datapoint, mind if I fold it into the model card as community-reported perf?
On the two issues, both reproduce on my side and I don't think either is (primarily) quant sensitivity:
Looping — likely the tokenizer EOG metadata, not the Q3. These GGUFs load with special_eos_id is not in special_eog_ids — the tokenizer config may be incorrect, and I can watch <｜hy_eos:opensource｜> leak into content on my cluster: the EOS token isn't registered as an end-of-generation token, so under some sampling/stop configs generation blows through it and keeps going. I'm investigating a metadata fix (would only touch first shards, like the arch patch below). Short-term: --jinja + an explicit stop on <｜hy_eos:opensource｜> kills most of it for me.
Tool calls — I can reproduce, and it's the native tools path specifically. Hy3 emits a bespoke tool-call format (<tool_calls:opensource> ... </tool_calls:opensource>, see the chat template), and llama-server's parser doesn't handle it — on my cluster a plain OpenAI-style tools request 500s with "The model produced output that does not match the expected peg-native format". So any harness using the native tools API will see frequent/total failures. Workaround that verifiably works: prompt-injected tools (describe tools in the prompt, parse the reply client-side — e.g. Continue.dev without the native tool_use capability drives browser automation fine with these quants). The model calls tools well; it's the server-side format parser that's missing on both PR builds.
Heads-up before you update llama.cpp: PR #25364 was closed; the successor PR #25395 (which adds working MTP spec decode — I measured +27% tg, 90% draft acceptance on a GB10) renamed the arch string hy-v3 → hy_v3. Your current files predate the rename, so they'll fail on the new build with unknown model architecture: 'hy-v3'. Fix is cheap: re-download only the first shard of your quant (repo was re-uploaded with the fix on 07-07), or run the 10-line in-place patch script now in the README.
Also, with 190GB across the pair you have room for IQ4_XS (159GB) — meaningfully better than Q3_K_M at ~4.3 BPW and my pick at your VRAM. If you rerun after switching to PR #25395, --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.75 --parallel 1 should stack nicely on those H100s (p_min matters — default makes spec a net loss on this model).
If you can share your serving flags / harness + a looping transcript, I'll dig further — genuinely useful reports, thanks.