README.md

metadata

language:
  - en
license: other
base_model:
  - Qwen/Qwen3.6-27B
tags:
  - gguf
  - llama.cpp
  - qwen
  - mtp
  - speculative-decoding
  - quantized
pipeline_tag: text-generation

Qwen3.6-27B-Claude-Opus-Sonnet-DistilledV2-MTP-GGUF

This is the GGUF quantized release of the local distilled model Qwen3.6-27B-Claude-Opus-Sonnet-DistilledV2-MTP.

The value proposition of this release is straightforward: it preserves the Claude Opus / Sonnet distilled style, opens MTP directly in llama.cpp for real acceleration, shortens the overly long reasoning chain seen in the local original model, and converts more of the token budget into user-visible answers.

Key points:

• Preserves the Claude Opus / Sonnet distilled response style and organization
• Verified to open MTP directly in llama.cpp with --spec-type draft-mtp
• Q4_K_M + MTP2 reaches 80.33% draft acceptance and 114.78 tok/s generation, versus 69.98 tok/s for Q4_K_M + non-MTP, or about 64% faster generation
• Compared with the local original model, this release follows a shorter reasoning path; in the same-machine 4-prompt comparison, the original consumed 9002 hidden reasoning chars
• Delivers higher visible-output efficiency per token budget; the same comparison produced 2845 visible answer chars for this release versus 1336 for the original
• Provides four quantization variants: Q2_K / Q4_K_M / Q6_K / Q8_0

1. Core Value Of This Release

This is not just a generic GGUF export. It is a release that has already been validated for local deployment. From an end-user perspective, the important points are:

• MTP can be opened directly in llama.cpp, rather than existing only as metadata that fails at runtime
• In the tested stack, MTP2 reaches 80.33% acceptance, showing that speculative acceleration is actually effective
• Same-machine comparison against the local original qwen3.6-27b shows that the original spends more of its budget on an overly long hidden reasoning chain
• This release turns more of the token budget into visible answers, making it better suited for efficient local deployment and interactive use

2. Files

File	Size	Notes
Qwen3.6-27B-Claude-Opus-Sonnet-DistilledV2-MTP-Q2_K.gguf	10.12 GB	Most aggressive compression, fastest, largest quality loss
Qwen3.6-27B-Claude-Opus-Sonnet-DistilledV2-MTP-Q4_K_M.gguf	15.66 GB	Best overall balance, default recommendation
Qwen3.6-27B-Claude-Opus-Sonnet-DistilledV2-MTP-Q6_K.gguf	20.89 GB	More quality-oriented, still reasonably fast
Qwen3.6-27B-Claude-Opus-Sonnet-DistilledV2-MTP-Q8_0.gguf	27.05 GB	Closer to high precision, heavier bandwidth pressure

3. Compatibility

Verified with:

• Windows CUDA build of llama.cpp
• GPU: NVIDIA RTX PRO 6000 Blackwell Workstation Edition 96 GB
• llama-cli
• --spec-type draft-mtp
• --spec-draft-n-max 2
• -ngl 999

Note: you need a newer llama.cpp build that includes Qwen3.5/3.6 MTP support. Older conversion pipelines may miss the required metadata and fail with failed to create MTP context.

4. Recommended Variant

• Q4_K_M: default recommendation, best speed/quality balance
• Q6_K: recommended if you care more about quality
• Q2_K: use when VRAM or disk space is very limited
• Q8_0: use for higher-fidelity experiments, but it is not always faster

5. GPU + MTP2 Benchmarks

Test environment:

• GPU: RTX PRO 6000 Blackwell 96 GB
• Backend: CUDA
• Args: -ngl 999 --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-ngl 999
• Logic puzzle: three-person truth/lie reasoning task, n=160
• 5.2 is the short-context benchmark
• 5.3 is the long-context benchmark

5.1 Historical Reference

Variant	Prompt	Generation	Draft acceptance
BF16 + MTP2	20.49 tok/s	0.85 tok/s	76.80%
Q4_K_M + non-MTP	796.22 tok/s	69.98 tok/s	-
Q4_K_M + MTP2	240.55 tok/s	114.78 tok/s	80.33%
Q4_K_M + MTP3	390.77 tok/s	117.16 tok/s	69.48%

5.2 Current Quantization Comparison

Variant	Prompt	Generation	Draft acceptance	Notes
Q2_K + MTP2	439.73 tok/s	118.01 tok/s	68.66%	Fastest generation, but most aggressive compression
Q4_K_M + MTP2	240.55 tok/s	114.78 tok/s	80.33%	Default recommendation
Q6_K + MTP2	503.87 tok/s	99.85 tok/s	78.86%	More quality-oriented
Q8_0 + MTP2	421.04 tok/s	78.86 tok/s	69.17%	Largest file, more bandwidth-limited

5.3 Long-Context Addendum

The long-context tests also use GPU + MTP2, but the prompt is changed to a long-document retrieval task:

• ctx8k uses an actual prompt length of about 6616 tokens
• ctx32k uses an actual prompt length of about 26738 tokens
• To reduce output variance, generation is intentionally short; the model usually reaches EOS after 17-23 tokens
• The table below is based on raw llama.cpp timing logs

Tier	Variant	Prompt tokens	Prompt tok/s	Generation tokens	Generation tok/s	Draft acceptance
ctx8k	Q2_K	6616	1304.11	16	104.41	83.33%
ctx8k	Q4_K_M	6616	2798.63	21	31.73	60.00%
ctx8k	Q6_K	6616	2415.74	21	69.48	60.00%
ctx8k	Q8_0	6616	2143.06	21	63.78	60.00%
ctx32k	Q2_K	26738	2450.46	17	71.41	78.57%
ctx32k	Q4_K_M	26738	2846.65	23	87.42	83.33%
ctx32k	Q6_K	26738	2620.59	17	81.02	71.43%
ctx32k	Q8_0	26738	3120.27	17	71.19	71.43%

Long-context observations:

• Q4_K_M remains the most balanced option in this long-context setup
• Q6_K still delivers 81 tok/s generation at ctx32k, making it a good quality-first choice
• Q8_0 shows strong prompt throughput at ctx32k, but generation still does not clearly outperform Q6_K
• Q2_K remains usable for long context, but it is still better suited for extreme compression than for default distribution

Conclusion:

• On this Blackwell workstation GPU, Q4_K_M remains the best-balanced variant
• Q2_K has the highest generation speed, but it is also the most aggressive in compression and quality trade-off
• Q6_K is more stable in acceptance and is a better high-quality option
• Q8_0 is not guaranteed to be faster, indicating clear bandwidth limits in this setup

5.4 Same-Machine Deployment Comparison vs Local Original qwen3.6-27b

This section presents a same-machine deployment comparison against the local original qwen3.6-27b served on port 1234. The purpose is to illustrate response efficiency, correctness, and output-budget allocation under a local deployment workflow, rather than to claim a strict cross-hardware or cross-framework benchmark result.

• Comparison target: qwen3.6-27b on http://127.0.0.1:1234/v1/chat/completions
• Release representative: Q4_K_M + MTP2
• Prompt set: 4 mostly objective prompts covering logic, a sqrt(2) proof, literary recall, and long-context retrieval
• Measurement note: the GGUF latency in the figure below includes llama-cli cold start, so this is a conservative comparison for the release

Key numbers:

• Average wall time: release 10.09s, original 10.93s
• Correctness: release 4/4, original 3/4
• Original hidden reasoning overhead: 9002 reasoning_content characters across the 4 prompts
• Release throughput: average prompt 1035.1 tok/s, average generation 118.1 tok/s

Observations:

• On this prompt set, the release is faster on average even though the GGUF side is measured with a cold start every run
• On the sqrt(2) proof prompt, the original spent a large amount of budget on hidden reasoning and did not reliably finish the final concise answer within the configured limit
• The release follows a direct-answer path and is better aligned with the goal of efficient local deployment
• If the release is deployed as a persistent local service instead of starting llama-cli per request, latency is typically lower than what is shown here

This comparison is not meant to be a formal academic benchmark. It answers a more practical question: on the same machine, can a publishable local GGUF release preserve correctness while delivering better response efficiency? In this test, the answer is yes.

6. Quality Validation

Two kinds of validation were performed:

1. Loadability validation
   • Q2_K / Q4_K_M / Q6_K / Q8_0 all passed the GGUF header check
   • All four quantization variants can be loaded with GPU draft-mtp
2. Same-prompt logic validation
   • Q2_K / Q4_K_M / Q6_K / Q8_0 all follow the same reasoning direction on the same truth/lie logic puzzle
   • The stable answer is:
     • A is lying
     • B is telling the truth
     • C is lying

Quality assessment:

Variant	Quality conclusion	Recommendation
Q2_K	Usable, but the most aggressive compression with the largest quality loss	Only recommended for extreme compression scenarios
Q4_K_M	Best overall balance of quality and speed	Default recommendation for release
Q6_K	More stable quality, better for fidelity-oriented use	Recommended as the higher-quality option
Q8_0	Quality is fine, but speed is not necessarily better than Q6_K	Recommended for high-fidelity experiments

Additional notes:

• In the current PowerShell + CLI environment, passing Chinese prompts directly via command-line arguments may occasionally introduce encoding noise
• Therefore, the main quality comparison in this repo uses an English logic puzzle as the unified benchmark
• For actual Chinese usage, validation through UTF-8 prompt files, API calls, or your own inference service is recommended

7. llama.cpp Usage

7.1 Regular Inference

./llama-cli \
  -m Qwen3.6-27B-Claude-Opus-Sonnet-DistilledV2-MTP-Q4_K_M.gguf \
  -ngl 999 \
  -c 8192 \
  -p "Introduce yourself briefly in Chinese."

7.2 Enable MTP

./llama-cli \
  -m Qwen3.6-27B-Claude-Opus-Sonnet-DistilledV2-MTP-Q4_K_M.gguf \
  -ngl 999 \
  -c 8192 \
  --spec-type draft-mtp \
  --spec-draft-n-max 2 \
  --spec-draft-ngl 999 \
  -p "Explain briefly how MTP works."

7.3 Recommended Args

Short replies:

-c 4096 --temp 0 --top-k 1 --spec-draft-n-max 2

Long reasoning:

-c 8192 --temp 0 --top-k 1 --spec-draft-n-max 2

MTP3 can still improve speed in some longer-output cases, but acceptance tends to drop. MTP2 is the recommended starting point.

8. Known Limitations

• Requires a newer llama.cpp
• BF16/Q4 exported by older converters may miss the key Qwen3.5/3.6 MTP metadata
• Some Windows CLI environments may corrupt Chinese prompt arguments
• Q8_0 is not guaranteed to be faster than Q6_K, especially on bandwidth-limited GPUs

9. Final Recommendation

If you only want to download one file:

• Choose Q4_K_M

If you care more about quality:

• Choose Q6_K

If you care more about extreme compression:

• Choose Q2_K

If you are running higher-fidelity experiments:

• Choose Q8_0