SC117

/

Ornith-1.0-35B-MTP-APEX-GGUF

like 80

I tested this model (I-Compact) using llama.ccp's speed-bench, and I'm not seeing any speedup from MTP. The same test with byteshape/Qwen3.6-35B-A3B-MTP-GGUF:IQ4_XS-3.53bpw shows a 30% speedup.

python llama.cpp/tools/server/bench/speed-bench/speed_bench.py \
                                   --url localhost:8080 \
                                   --bench qualitative \
                                   --category coding \
                                   --limit 8 \
                                   --osl 1024 \
                                   --concurrency 1 \
                                   --output ornith_nothinking_coding_draft-mtp_3.json

This was run against llama-server with and without MTP enabled:

llama-server -hf SC117/Ornith-1.0-35B-MTP-APEX-GGUF:Ornith-1.0-35B-MTP-APEX-I-Compact \
  -b 2048 -ub 2048 \
  --ctx-size 8192 \
  --temp 1.0 --top-p 1.0 --top-k 0 \
  --jinja --chat-template-file Qwen-Fixed-Chat-Templates/chat_template.jinja \
  -np 1 \
  --reasoning on/off \
 [--spec-type draft-mtp --spec-draft-n-max 3]

category  base_avg_pred_t/s  spec_avg_pred_t/s  decode_speedup  base_avg_latency  spec_avg_latency  latency_speedup  accept_rate
--------  -----------------  -----------------  --------------  ----------------  ----------------  ---------------  -----------
coding    54.13              53.86              1.00x           21.842s           21.683s           1.01x            0.6662  # --reasoning on
coding    56.31              48.57              0.86x           11.720s           13.695s           0.86x            0.8025  # --reasoning off

Did you check the effectiveness of MTP in this model?

Thanks for the benchmark.

A clarification about the MTP implementation:

The MTP layers are sourced from the original Qwen3.5-35B-A3B model because Ornith is trained on the same Qwen3.5-35B-A3B architecture, making the MTP weights structurally compatible.

However, this is different from a model where the MTP head was jointly trained together with the final model weights. Therefore, the MTP acceptance rate and speedup may not necessarily match the original Qwen3.5/Qwen3.6 MTP models.

The most meaningful comparison would be Ornith without MTP vs Ornith with MTP enabled under identical runtime settings.

One more important note: MTP requires additional memory because the MTP head also needs to be loaded.

For users who are already close to the VRAM limit, enabling MTP may force additional CPU offloading, and the resulting offload overhead can cancel out the speculative decoding speedup.

For the best MTP performance, it is recommended to run with enough VRAM to keep the model (including the MTP head) fully on GPU without CPU offloading.

Thanks for helping test this!

I tested this model (I-Compact) using llama.ccp's speed-bench, and I'm not seeing any speedup from MTP. The same test with byteshape/Qwen3.6-35B-A3B-MTP-GGUF:IQ4_XS-3.53bpw shows a 30% speedup.

python llama.cpp/tools/server/bench/speed-bench/speed_bench.py \
                                   --url localhost:8080 \
                                   --bench qualitative \
                                   --category coding \
                                   --limit 8 \
                                   --osl 1024 \
                                   --concurrency 1 \
                                   --output ornith_nothinking_coding_draft-mtp_3.json

This was run against llama-server with and without MTP enabled:

llama-server -hf SC117/Ornith-1.0-35B-MTP-APEX-GGUF:Ornith-1.0-35B-MTP-APEX-I-Compact \
  -b 2048 -ub 2048 \
  --ctx-size 8192 \
  --temp 1.0 --top-p 1.0 --top-k 0 \
  --jinja --chat-template-file Qwen-Fixed-Chat-Templates/chat_template.jinja \
  -np 1 \
  --reasoning on/off \
 [--spec-type draft-mtp --spec-draft-n-max 3]

category  base_avg_pred_t/s  spec_avg_pred_t/s  decode_speedup  base_avg_latency  spec_avg_latency  latency_speedup  accept_rate
--------  -----------------  -----------------  --------------  ----------------  ----------------  ---------------  -----------
coding    54.13              53.86              1.00x           21.842s           21.683s           1.01x            0.6662  # --reasoning on
coding    56.31              48.57              0.86x           11.720s           13.695s           0.86x            0.8025  # --reasoning off

Did you check the effectiveness of MTP in this model?

Thanks for testing and reporting the results.

I also tested the same GGUF with and without MTP enabled on my side. Here are my results:

Hardware:

• AMD AI Max+ 395
• 128GB unified memory

Runtime:

• llama.cpp (MTP enabled)
• Same model and same runtime settings for both tests
• --spec-type draft-mtp --spec-draft-n-max 2

MTP enabled:

code_python pred=192 draft=138 acc=121 rate=0.877 tok/s=80.4
code_cpp pred= 53 draft= 38 acc= 34 rate=0.895 tok/s=78.2
explain_concept pred=192 draft=165 acc=108 rate=0.654 tok/s=68.3
summarize pred= 49 draft= 38 acc= 30 rate=0.789 tok/s=72.7
qa_factual pred=192 draft=146 acc=117 rate=0.801 tok/s=69.5
translation pred= 17 draft= 14 acc= 10 rate=0.714 tok/s=58.6
creative_short pred= 36 draft= 42 acc= 15 rate=0.357 tok/s=44.2
stepwise_math pred=192 draft=138 acc=121 rate=0.877 tok/s=71.9
long_code_review pred=192 draft=185 acc= 97 rate=0.524 tok/s=49.3

Aggregate:
total_predicted: 1115
total_draft: 904
total_draft_accepted: 653
aggregate_accept_rate: 0.7223
wall_s_total: 19.83

MTP disabled:

code_python tok/s=60.4
code_cpp tok/s=59.6
explain_concept tok/s=59.5
summarize tok/s=55.7
qa_factual tok/s=57.3
translation tok/s=56.0
creative_short tok/s=56.0
stepwise_math tok/s=57.1
long_code_review tok/s=56.1

Aggregate:
total_predicted: 1123
wall_s_total: 22.08

In this test, MTP improved the overall throughput from about 50.9 tok/s to 56.2 tok/s (~10% faster), with an average acceptance rate of 72.2%.

A few things may affect the final MTP gain:

• MTP adds additional memory usage because the MTP head also needs to be loaded.
• If enabling MTP causes additional CPU offloading, the offload overhead may cancel out the speculative decoding benefit.
• Different hardware, llama.cpp versions, quantization formats and --spec-draft-n-max values can result in different performance.

Also, Ornith's MTP head is sourced from Qwen3.5-35B-A3B because Ornith is trained on the same Qwen3.5-35B-A3B architecture, so the MTP weights are structurally compatible. However, direct comparisons with other models' native MTP implementations may not be fully equivalent.

Thanks again for helping test the MTP performance!