Upload README.md with huggingface_hub

README.md

@@ -54,11 +54,13 @@ On low-bit IQ tiers the MTP/NextN layer (`blk.80.*`) is kept at q8_0 (`--tensor-
 
 Single node, fully GPU-resident, `-fa on`:
 
-| Quant | Gen tok/s | Prompt tok/s |
-|---|---|---|
-| IQ2_M | 18.5 | 36.4 |
-| Q2_K | 18.1 | 35.5 |
-| IQ3_XXS | 17.1 | 32.4 |
+| Quant | Gen tok/s | + MTP spec decode | Prompt tok/s |
+|---|---|---|---|
+| IQ2_M | 18.0–18.5 | **22.8 (+27%)** | 36–50 |
+| Q2_K | 18.1 | untested | 35.5 |
+| IQ3_XXS | 17.1 | untested | 32.4 |
+
+MTP numbers measured at temp 0 with `--spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.75` (90% draft acceptance). Higher sampling temperatures reduce acceptance and land between the two columns.
 
 Dual node (Q4_K_M, 181GB layer-split across 2× GB10 over 200GbE via llama.cpp RPC): **14.0 tok/s gen / 21.9 tok/s prompt**. RPC layer-split adds capacity for bigger quants, not speed — expect single-node-or-slower decode rates.
 
@@ -86,7 +88,34 @@ Tips:
 
 ## MTP / speculative decoding status
 
-Hy3's MTP (multi-token prediction) tensors are **converted and stored** in these GGUFs, but PR #25364 currently skips them in the forward graph — llama.cpp **cannot yet use them for speculative decoding**. The weights are preserved (q8_0 on low-bit tiers) so existing files become spec-decode-ready if/when the PR adds graph support. No re-download should be needed.
+**Verified working (2026-07-07):** [PR #25395](https://github.com/ggml-org/llama.cpp/pull/25395) adds Hy3 MTP speculative decoding, and these GGUFs work with it as-is — the MTP tensors bundled in every quant (q8_0-preserved on low-bit tiers) are used directly as the `draft-mtp` target. Measured on IQ2_M: **18.0 → 22.8 tok/s (+27%), 90% draft acceptance**.
+
+```bash
+./build/bin/llama-server -m Hy3-IQ2_M/Hy3-IQ2_M-00001-of-00003.gguf \
+  -ngl 99 -fa on -c 32768 --jinja \
+  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-p-min 0.75 \
+  --parallel 1
+```
+
+Notes:
+- `--spec-draft-p-min 0.75` is **required for a speedup** — the MTP head is trained single-depth, and the default p_min makes speculation a net loss (per the PR author's measurements, confirmed here).
+- `--parallel 1` is required for draft-mtp; `n_max` 2 and 3 measure within ~1% of each other (n=2 slightly ahead at 90% acceptance).
+- `--spec-type` exists on `llama-server` and `llama-cli` only, not `llama-completion`.
+
+### ⚠️ If you downloaded before 2026-07-08: arch string fix
+
+PR #25395 renamed the architecture string from `hy-v3` (the earlier PR #25364) to `hy_v3`. All first-shards in this repo were re-uploaded with the fix on 2026-07-07, so fresh downloads just work. If you hold older files and see `unknown model architecture: 'hy-v3'`, either re-download the first shard of your quant, or patch in place (the string lives only in shard 00001's header; byte-for-byte same length):
+
+```python
+# python patch_arch.py <your-first-shard.gguf>  — swaps hy-v3 -> hy_v3 in the header
+import mmap, sys
+with open(sys.argv[1], "r+b") as f:
+    mm = mmap.mmap(f.fileno(), 64 * 1024 * 1024)  # metadata lives well within 64MB
+    i = mm.find(b"hy-v3")
+    while i != -1:
+        mm[i:i+5] = b"hy_v3"; i = mm.find(b"hy-v3", i + 1)
+    mm.flush()
+```
 
 ## Provenance