README.md

metadata

language:
  - en
  - zh
base_model:
  - Qwen/Qwen3.6-27B
tags:
  - gguf
  - llama.cpp
  - qwen
  - qwen3
  - qwen3.6
  - mtp
  - speculative-decoding
  - yarn
  - long-context
  - 1m-context
  - imatrix
  - quantized
  - distillation
  - agent
  - react
  - 1M-context
pipeline_tag: text-generation
license: apache-2.0
library_name: gguf
inference:
  parameters:
    temperature: 0.6
    top_p: 0.95
model-index:
  - name: Opus-DeepSeek-Distilled-Q4M
    results:
      - task:
          type: text-generation
        dataset:
          type: benchlocal
          name: BenchLocal 6-pack
        metrics:
          - type: benchlocal-score
            value: 86.5
            name: 6-pack Total
      - task:
          type: question-answering
        dataset:
          type: gpqa
          name: GPQA-Diamond-198
        metrics:
          - type: accuracy
            value: 83.84
            name: Accuracy
      - task:
          type: question-answering
        dataset:
          type: mmlu
          name: MMLU-500 (5-shot)
        metrics:
          - type: accuracy
            value: 91.8
            name: Accuracy

🧠 Qwen3.6-27B-Claude-Opus-DeepSeek-Distilled — YaRN 1M Context

A distilled Qwen3.6-27B GGUF with YaRN 4× context extension to 1,048,576 tokens, optimized for local agentic reasoning, tool use, and long-chain task execution.

---

⚠️ Sampling Parameters

Please ensure temperature = 0.6 and top_p = 0.95 when using this model.

This model was trained and validated at these specific parameters. Both too high and too low temperatures cause problems:

🔥 Too high (> 0.6) → Output becomes divergent as token probabilities flatten. This causes malformed tool calls, function name hallucinations, unstable parameter generation, and uncontrollable agent behavior.

🧊 Too low (< 0.3) → The model almost always picks the highest-probability token. In tool-calling scenarios this manifests as: repeating the same failed tool call instead of trying alternatives, shortened thinking traces leading to insufficient reasoning depth, and reduced robustness to edge cases.

This is an important recommendation based on extensive real-world testing. Please verify these parameters in your inference framework.

---

📢 Highlights

Area	Score	vs Qwen3.6-27B q4_k_m
BenchLocal 6-pack 🏆	86.5	+8.3
GPQA-Diamond-198 🔬	83.84%	+10.14%
BugFind-15 🐛	80	+20
ToolCall-15 🔧	97	+4
InstructFollow-15 📋	94	+17
StructOutput-15 📊	88	+11
MMLU-500 (5-shot) 📚	91.80%	~tied (+0.2%)
DataExtract-15 📄	81	-2
Context window 🏛️	1,048,576	4× (262K → 1M)

🚀 Output speed: ~60 tok/s on A100 40GB · ~100 tok/s on RTX PRO 6000 (q4_k_m + mtp=3)

---

🔥 Why This Model?

The original Qwen3.6-27B has solid foundational capabilities, but its agent behavior falls short — prone to infinite loops when thinking, lacks structured agent design, and has room to improve in math reasoning.

This variant tackles all three through targeted distillation:

1. ✅ Infinite loops → Eliminated. ReAct-style reasoning-action orchestration fixes the root cause.
2. ✅ Agent behavior → Structured. Distilled Claude Opus's systematic thinking and organization.
3. ✅ Math reasoning → Strengthened. Absorbed capabilities from strong math/logic models.

Additionally, all quantized variants in this repo include YaRN 4× context extension (native 262K → 1,048,576 tokens), enabling processing of book-length documents, long codebases, and extended multi-turn conversations.

The result is a local agent that doesn't just score high on benchmarks — it works reliably in real engineering tasks like BugFind, and now handles 1 million tokens of context.

⚠️ Note: Side-by-side GLM5.2 comparisons were evaluated using Opus 4.8 as a judge. Opus-as-judge has inherent biases — results are indicative, not definitive.

---

🎯 Design Philosophy

Core insight: Qwen3.6-27B doesn't lack capability — it lacks good agent behavior. That makes it worth iterating on.

We followed the ReAct paradigm (Yao et al., ICLR 2023) — unifying reasoning and action into an alternating, constrained, executable loop — rather than just making the model "think longer" or "call tools better."

Teacher Model	Capability Distilled
Claude Opus 🎯	Systematic thinking, structured organization, concise reasoning
DeepSeek 🧭	Stable agent behavior, tool orchestration, task closure
Math/Logic models ➗	Mathematical reasoning, logical deduction

Context Extension

The 1M context window is achieved via YaRN (Yet another RoPE extensioN) method (Peng et al., 2023), a state-of-the-art approach for extending LLM context windows without fine-tuning. YaRN adjusts the rotary position encoding (RoPE) by interpolating frequencies and applying a neural tangent kernel (NTK)-aware scaling, enabling 4× extension from the native 262K to 1,048,576 tokens while preserving the model's existing capabilities.

---

📊 Performance

BenchLocal 6-pack

Pack	q4_k_m (ours)	Qwen/Qwen3.6-27B q4_k_m	Delta
BugFind-15 🐛	80	60	+20
ToolCall-15 🔧	97	93	+4
DataExtract-15 📄	81	83	-2
InstructFollow-15 📋	94	77	+17
ReasonMath-15 ➗	79	79	0
StructOutput-15 📊	88	77	+11
Total 🏆	86.5	78.2	+8.3

Extended Evals

Benchmark	Ours	Baseline	Notes
GPQA-Diamond-198 🔬	83.84%	73.7%	+10.14%, all 198 graded locally
MMLU-500 (5-shot) 📚	91.80%	91.6%	Approximately tied

---

📦 Files

All GGUFs include YaRN 1M context scaling baked into the metadata — no extra flags needed.

File	Size	Quant
...GGUF-Q2_K-YaRN-1M.gguf	11 GB	Q2_K (imatrix)
...GGUF-Q3_K_M-YaRN-1M.gguf	13 GB	Q3_K_M (imatrix)
...GGUF-Q4_K_M-YaRN-1M.gguf	16 GB	Q4_K_M (imatrix)
...GGUF-Q5_K_M-YaRN-1M.gguf	19 GB	Q5_K_M (imatrix)
...GGUF-Q6_K-YaRN-1M.gguf	21 GB	Q6_K (imatrix)

YaRN Parameters Embedded

Parameter	Value
rope.scaling.type	yarn
rope.scaling.factor	4.0
rope.scaling.original_context_length	262144
rope.scaling.yarn_ext_factor	1.0
rope.scaling.yarn_attn_factor	1.0
rope.scaling.yarn_beta_fast	32.0
rope.scaling.yarn_beta_slow	1.0
context_length	1048576

---

🛠️ Usage

Recommended Stack

🧩 OpenCode + LM Studio / llama.cpp
📐 Temperature: 0.6 · Top-p: 0.95
⚡ q4_k_m + mtp=3
🧠 Context: -c 1048576

Quick Start (llama.cpp)

# Download (choose your quantization)
huggingface-cli download Brian6145/Qwen3.6-27B-Claude-Opus-DeepSeek-Distilled-Imatrix-MTP-1M-GGUF \
  <QUANT>-YaRN-1M.gguf --local-dir ./models

# Run with 1M context — YaRN params baked in
./llama-cli -m ./models/<QUANT>-YaRN-1M.gguf \
  --temp 0.6 --top-p 0.95 \
  -c 1048576 \
  -p "Your prompt here"

# Example with Q4_K_M:
./llama-cli -m ./models/Qwen3.6-27B-Claude-Opus-DeepSeek-Distilled-Imatrix-MTP-1M-GGUF-Q4_K_M-YaRN-1M.gguf \
  --temp 0.6 --top-p 0.95 \
  -c 1048576 \
  -p "Explain the entire codebase in this repository:"

⚠️ 1M context requires substantial memory. At Q4_K_M, KV cache alone is ~15-20 GB. A100-80GB or equivalent recommended. For smaller GPUs, reduce -c to 128K or 256K.

Via LM Studio

1. Load the GGUF file in LM Studio
2. Set backend to llama.cpp
3. Set context length to 1048576 (or your hardware limit)
4. Enable MTP (set depth=3) under inference options
5. Set temperature = 0.6, top_p = 0.95
6. Start the local API server
7. Connect via OpenCode or any OpenAI-compatible client

💡 Pro tip: For coding tasks, the temp 0.6 / top_p 0.95 combo delivers the best balance of creativity and correctness.

---

⚠️ Known Limitations

Evaluation Methodology

• Opus-as-judge biases: GLM5.2 comparisons are judge-evaluated, not absolute rankings
• MMLU: Single-run 5-shot result; variations in shot selection may cause fluctuation

Capability Boundaries

• DataExtract: Scores 81 vs 83 baseline — extraction tasks may have slight regression from distillation
• Closed-source teachers: Risks include inherited biases and TOS compliance — assess for your use case
• 27B scale ceiling: May still hit capacity limits on extremely complex long-chain reasoning

Context Extension

• YaRN is inference-only: The 1M context window is achieved through RoPE scaling, not fine-tuning. Performance at extreme lengths may degrade gracefully rather than maintain full fidelity
• Hardware requirements: 1M KV cache at Q4_K_M requires ~15-20 GB VRAM just for the cache, plus ~16 GB for model weights. A100-80GB or multi-GPU setup recommended

Deployment Notes

• MTP=3: Boosts throughput but adds VRAM overhead — disable or reduce on <24GB hardware
• Imatrix: Uses importance-matrix quantization, not standard k-quant — better parameter preservation at low bit widths
• All quants included: Unlike the base repo which only ships q4_k_m, this release includes Q2 through Q6 for flexibility

---

🔗 YaRN: Yet another RoPE extensioN Method

This release uses YaRN (Peng et al., 2023) to extend the native 262K context window to 1M tokens. YaRN is a state-of-the-art context extension method that:

• Preserves existing capabilities — no fine-tuning needed, the model retains all original knowledge
• Uses NTK-aware scaling — better allocation of frequency dimensions than linear interpolation
• Adds a temperature ramp — smooth transition between trained and extrapolated positions
• Achieves strong perplexity — outperforms linear and NTK-aware scaling at extended contexts

The parameters are baked directly into each GGUF file's metadata — llama.cpp reads them automatically at load time.

---

📝 Citation

@article{peng2023yarn,
  title = {YaRN: Efficient Context Window Extension of Large Language Models},
  author = {Bowen Peng and Jeffrey Quesnelle and Honglu Fan and Enrico Shippole},
  journal = {arXiv preprint arXiv:2305.13298},
  year = {2023}
}

@misc{opus-deepseek-distilled-q4m,
  title = {Opus-DeepSeek-Distilled-Q4M: A Distilled Agentic GGUF for Local Deployment},
  author = {Yin, Brian and BenchLocal Contributors},
  year = {2026},
  url = {https://github.com/brianyin/BenchLocal}
}

---

🙏 Acknowledgements

• Qwen team — excellent foundational model
• Unsloth — efficient training infrastructure
• Merkyor — identified ReAct as the key to solving agent infinite loops
• Community — built on existing open-source exploration and practical experience

It is because of this work that came before that we can continue pushing forward, arriving at today's more stable, more practical, and more complete agent — and helping us get closer to the era of local agent AI.