---
license: apache-2.0
language:
- en
base_model:
- Qwen/Qwen3.5-4B
---

# Usage
For now, you need to use the gg/spec-mtp-experiments branch on llama.cpp or a custom mtp fork.

You can switch to the mtp branch with `git checkout gg/spec-mtp-experiments` after cloning and entering the llama.cpp repository.

Add `--spec-type mtp --spec-draft-n-max 5 --spec-draft-n-min 0` to your llama-server or llama-cli command.

Feel free to tweak `--spec-draft-n-max` and find out what works best for your setup.

Try not to push `--spec-draft-n-min` too far, keep it in single digits.

I found that (in my testing), token speed was as such when tweaking `--spec-draft-n-min`:

| Setting for `--spec-draft-n-min` | generation t/s |
| --- | --- |
| 0 | 47.7 |
| 1 | 47.2 |
| 2 | 47.8 |
| 3 | 44.2 |
| 4 | 44.9 |
| 5 | 36.2 |

with the launch command being

```
/llama.cpp/build/bin/llama-cli -st -p 'What is the antiderivative of x^3?' --verbose-prompt --prio 3 --batch-size 1024 --ubatch-size 1024 --mmap --perf --flash-attn on --fit-ctx 16384 -ctk q8_0 -ctv q8_0 -m /ai/models/Qwen3.5-4B-Q5_K_M-mtp.gguf -ngl all --spec-type mtp --spec-draft-n-max 5 --spec-draft-n-min "$i" -fitt 2048
```

# Credits
- Qwen for this amazing model.
- Unsloth for the imatrix quantization file.
- All of the llama.cpp and ggml contributors for allowing me to run AI models locally.