README.md

metadata

language:
  - en
  - zh
base_model:
  - Qwen/Qwen3.6-27B
tags:
  - gguf
  - llama.cpp
  - qwen
  - qwen3
  - qwen3.6
  - mtp
  - speculative-decoding
  - quantized
  - long-context
  - chinese
  - distillation
  - agent
  - react
pipeline_tag: text-generation
license: apache-2.0
library_name: gguf
inference:
  parameters:
    temperature: 0.6
    top_p: 0.95
model-index:
  - name: Opus-DeepSeek-Distilled-Q4M
    results:
      - task:
          type: text-generation
        dataset:
          type: benchlocal
          name: BenchLocal 6-pack
        metrics:
          - type: benchlocal-score
            value: 86.5
            name: 6-pack Total
      - task:
          type: question-answering
        dataset:
          type: gpqa
          name: GPQA-Diamond-198
        metrics:
          - type: accuracy
            value: 83.84
            name: Accuracy
      - task:
          type: question-answering
        dataset:
          type: mmlu
          name: MMLU-500 (5-shot)
        metrics:
          - type: accuracy
            value: 91.8
            name: Accuracy

🧠 Opus-DeepSeek-Distilled-Q4M

A distilled Qwen3.6-27B GGUF optimized for local agentic reasoning, tool use, and long-chain task execution.

---

⚠️ Sampling Parameters

Please ensure temperature = 0.6 and top_p = 0.95 when using this model.

This model was trained and validated at these specific parameters. Both too high and too low temperatures cause problems:

🔥 Too high (> 0.6) → Output becomes divergent as token probabilities flatten. This causes malformed tool calls, function name hallucinations, unstable parameter generation, and uncontrollable agent behavior.

🧊 Too low (< 0.3) → The model almost always picks the highest-probability token. In tool-calling scenarios this manifests as: repeating the same failed tool call instead of trying alternatives, shortened thinking traces leading to insufficient reasoning depth, and reduced robustness to edge cases.

This is an important recommendation based on extensive real-world testing. Please verify these parameters in your inference framework.

---

📢 Highlights

Area	Score	vs Qwen3.6-27B q4_k_m
BenchLocal 6-pack 🏆	86.5	+8.3
GPQA-Diamond-198 🔬	83.84%	+10.14%
BugFind-15 🐛	80	+20
ToolCall-15 🔧	97	+4
InstructFollow-15 📋	94	+17
StructOutput-15 📊	88	+11
MMLU-500 (5-shot) 📚	91.80%	~tied (+0.2%)
DataExtract-15 📄	81	-2

🚀 Output speed: ~60 tok/s on A100 40GB · ~100 tok/s on RTX PRO 6000 (q4_k_m + mtp=3)

---

🔥 Why This Model?

The original Qwen3.6-27B has solid foundational capabilities, but its agent behavior falls short — prone to infinite loops when thinking, lacks structured agent design, and has room to improve in math reasoning.

This variant tackles all three through targeted distillation:

1. ✅ Infinite loops → Eliminated. ReAct-style reasoning-action orchestration fixes the root cause.
2. ✅ Agent behavior → Structured. Distilled Claude Opus's systematic thinking and organization.
3. ✅ Math reasoning → Strengthened. Absorbed capabilities from strong math/logic models.

The result is a local agent that doesn't just score high on benchmarks — it works reliably in real engineering tasks like BugFind, where it now competes with GLM5.2.

⚠️ Note: Side-by-side GLM5.2 comparisons were evaluated using Opus 4.8 as a judge. Opus-as-judge has inherent biases — results are indicative, not definitive.

---

🎯 Design Philosophy

Core insight: Qwen3.6-27B doesn't lack capability — it lacks good agent behavior. That makes it worth iterating on.

We followed the ReAct paradigm (Yao et al., ICLR 2023) — unifying reasoning and action into an alternating, constrained, executable loop — rather than just making the model "think longer" or "call tools better."

Teacher Model	Capability Distilled
Claude Opus 🎯	Systematic thinking, structured organization, concise reasoning
DeepSeek 🧭	Stable agent behavior, tool orchestration, task closure
Math/Logic models ➗	Mathematical reasoning, logical deduction

---

📊 Performance

BenchLocal 6-pack

Pack	q4_k_m (ours)	Qwen/Qwen3.6-27B q4_k_m	Delta
BugFind-15 🐛	80	60	+20
ToolCall-15 🔧	97	93	+4
DataExtract-15 📄	81	83	-2
InstructFollow-15 📋	94	77	+17
ReasonMath-15 ➗	79	79	0
StructOutput-15 📊	88	77	+11
Total 🏆	86.5	78.2	+8.3

Extended Evals

Benchmark	Ours	Baseline	Notes
GPQA-Diamond-198 🔬	83.84%	73.7%	+10.14%, all 198 graded locally
MMLU-500 (5-shot) 📚	91.80%	91.6%	Approximately tied

---

🛠️ Usage

Recommended Stack

🧩 OpenCode + LM Studio
📐 Temperature: 0.6 · Top-p: 0.95
⚡ q4_k_m + mtp=3

Quick Start (llama.cpp)

# Download the GGUF
huggingface-cli download your-org/Opus-DeepSeek-Distilled-Q4M \
  opus-deepseek-distilled-q4m-q4_k_m.gguf --local-dir ./models

# Run with llama.cpp
./llama-cli -m ./models/opus-deepseek-distilled-q4m-q4_k_m.gguf \
  --temp 0.6 --top-p 0.95 \
  -p "Your prompt here"

Via LM Studio

1. Load the GGUF file in LM Studio
2. Set backend to llama.cpp
3. Enable MTP (set depth=3) under inference options
4. Set temperature = 0.6, top_p = 0.95
5. Start the local API server
6. Connect via OpenCode or any OpenAI-compatible client

💡 Pro tip: For coding tasks, the temp 0.6 / top_p 0.95 combo delivers the best balance of creativity and correctness.

---

⚠️ Known Limitations

Evaluation Methodology

• Opus-as-judge biases: GLM5.2 comparisons are judge-evaluated, not absolute rankings
• MMLU: Single-run 5-shot result; variations in shot selection may cause fluctuation

Capability Boundaries

• DataExtract: Scores 81 vs 83 baseline — extraction tasks may have slight regression from distillation
• Closed-source teachers: Risks include inherited biases and TOS compliance — assess for your use case
• 27B scale ceiling: May still hit capacity limits on extremely complex long-chain reasoning

Deployment Notes

• MTP=3: Boosts throughput but adds VRAM overhead — disable or reduce on <24GB hardware
• Quantization: Only q4_k_m tested; q5_k_m may improve accuracy at higher VRAM cost
• Imatrix: Uses importance-matrix quantization, not standard k-quant — better parameter preservation at low bit widths

---

📝 Citation

@misc{opus-deepseek-distilled-q4m,
  title = {Opus-DeepSeek-Distilled-Q4M: A Distilled Agentic GGUF for Local Deployment},
  author = {Yin, Brian and BenchLocal Contributors},
  year = {2026},
  url = {https://github.com/brianyin/BenchLocal}
}

---

🙏 Acknowledgements

• Qwen team — excellent foundational model
• Unsloth — efficient training infrastructure
• Merkyor — identified ReAct as the key to solving agent infinite loops
• Community — built on existing open-source exploration and practical experience

It is because of this work that came before that we can continue pushing forward, arriving at today's more stable, more practical, and more complete agent — and helping us get closer to the era of local agent AI.