README.md

---
language:
  - en
  - zh
base_model:
  - Qwen/Qwen3.6-27B
tags:
  - gguf
  - llama.cpp
  - qwen
  - qwen3
  - qwen3.6
  - mtp
  - speculative-decoding
  - yarn
  - long-context
  - 1m-context
  - imatrix
  - quantized
  - distillation
  - agent
  - react
  - 1M-context
pipeline_tag: text-generation
license: apache-2.0
library_name: gguf
inference:
  parameters:
    temperature: 0.6
    top_p: 0.95
model-index:
  - name: Opus-DeepSeek-Distilled-Q4M
    results:
      - task:
          type: text-generation
        dataset:
          type: benchlocal
          name: BenchLocal 6-pack
        metrics:
          - type: benchlocal-score
            value: 86.5
            name: 6-pack Total
      - task:
          type: question-answering
        dataset:
          type: gpqa
          name: GPQA-Diamond-198
        metrics:
          - type: accuracy
            value: 83.84
            name: Accuracy
      - task:
          type: question-answering
        dataset:
          type: mmlu
          name: MMLU-500 (5-shot)
        metrics:
          - type: accuracy
            value: 91.80
            name: Accuracy
---

# 🧠 Qwen3.6-27B-Claude-Opus-DeepSeek-Distilled — YaRN 1M Context

> A distilled Qwen3.6-27B GGUF with **YaRN 4× context extension** to 1,048,576 tokens, optimized for local agentic reasoning, tool use, and long-chain task execution.

[![BenchLocal](https://img.shields.io/badge/BenchLocal-86.5-8A2BE2)](https://github.com/brianyin/BenchLocal)
[![GPQA](https://img.shields.io/badge/GPQA--Diamond--198-83.84%25-blue)](https://github.com/brianyin/BenchLocal)
[![MMLU](https://img.shields.io/badge/MMLU--500-91.80%25-green)](https://github.com/brianyin/BenchLocal)
[![License](https://img.shields.io/badge/License-Apache--2.0-orange)](https://www.apache.org/licenses/LICENSE-2.0)
[![GGUF](https://img.shields.io/badge/Format-GGUF-ff69b4)](https://github.com/ggml-org/llama.cpp)
[![Context](https://img.shields.io/badge/Context-1M-8A2BE2)](https://arxiv.org/abs/2305.13298)

---

## ⚠️ Sampling Parameters

> **Please ensure `temperature = 0.6` and `top_p = 0.95` when using this model.**
>
> This model was trained and validated at these specific parameters. Both too high and too low temperatures cause problems:
>
> 🔥 **Too high (> 0.6)** → Output becomes divergent as token probabilities flatten. This causes malformed tool calls, function name hallucinations, unstable parameter generation, and uncontrollable agent behavior.
>
> 🧊 **Too low (< 0.3)** → The model almost always picks the highest-probability token. In tool-calling scenarios this manifests as: repeating the same failed tool call instead of trying alternatives, shortened thinking traces leading to insufficient reasoning depth, and reduced robustness to edge cases.
>
> This is an **important recommendation** based on extensive real-world testing. Please verify these parameters in your inference framework.

---

## 📢 Highlights

| Area | Score | vs Qwen3.6-27B q4_k_m |
|------|-------|----------------------|
| **BenchLocal 6-pack** 🏆 | **86.5** | **+8.3** |
| **GPQA-Diamond-198** 🔬 | **83.84%** | +10.14% |
| **BugFind-15** 🐛 | **80** | **+20** |
| **ToolCall-15** 🔧 | **97** | +4 |
| **InstructFollow-15** 📋 | **94** | **+17** |
| **StructOutput-15** 📊 | **88** | **+11** |
| **MMLU-500 (5-shot)** 📚 | **91.80%** | ~tied (+0.2%) |
| **DataExtract-15** 📄 | 81 | -2 |
| **Context window** 🏛️ | **1,048,576** | **4×** (262K → 1M) |

> 🚀 **Output speed**: `~60 tok/s` on A100 40GB · `~100 tok/s` on RTX PRO 6000 (q4_k_m + mtp=3)

---

## 🔥 Why This Model?

The original `Qwen3.6-27B` has solid foundational capabilities, but its **agent behavior falls short** — prone to infinite loops when thinking, lacks structured agent design, and has room to improve in math reasoning.

This variant tackles all three through targeted distillation:

1. ✅ **Infinite loops → Eliminated.** ReAct-style reasoning-action orchestration fixes the root cause.
2. ✅ **Agent behavior → Structured.** Distilled Claude Opus's systematic thinking and organization.
3. ✅ **Math reasoning → Strengthened.** Absorbed capabilities from strong math/logic models.

**Additionally**, all quantized variants in this repo include **YaRN 4× context extension** (native 262K → 1,048,576 tokens), enabling processing of book-length documents, long codebases, and extended multi-turn conversations.

The result is a local agent that doesn't just score high on benchmarks — it *works* reliably in real engineering tasks like BugFind, and now handles **1 million tokens of context**.

> ⚠️ **Note**: Side-by-side GLM5.2 comparisons were evaluated using Opus 4.8 as a judge. Opus-as-judge has inherent biases — results are indicative, not definitive.

---

## 🎯 Design Philosophy

**Core insight**: `Qwen3.6-27B` doesn't lack capability — it lacks *good agent behavior*. That makes it worth iterating on.

We followed the **[ReAct](https://arxiv.org/abs/2210.03629)** paradigm (Yao et al., ICLR 2023) — unifying reasoning and action into an alternating, constrained, executable loop — rather than just making the model "think longer" or "call tools better."

| Teacher Model | Capability Distilled |
|---------------|---------------------|
| **Claude Opus** 🎯 | Systematic thinking, structured organization, concise reasoning |
| **DeepSeek** 🧭 | Stable agent behavior, tool orchestration, task closure |
| **Math/Logic models** ➗ | Mathematical reasoning, logical deduction |

### Context Extension

The 1M context window is achieved via **YaRN (Yet another RoPE extensioN)** method ([Peng et al., 2023](https://arxiv.org/abs/2305.13298)), a state-of-the-art approach for extending LLM context windows without fine-tuning. YaRN adjusts the rotary position encoding (RoPE) by interpolating frequencies and applying a neural tangent kernel (NTK)-aware scaling, enabling **4× extension** from the native 262K to 1,048,576 tokens while preserving the model's existing capabilities.

---

## 📊 Performance

### BenchLocal 6-pack

| Pack | `q4_k_m` (ours) | `Qwen/Qwen3.6-27B q4_k_m` | Delta |
|------|:-:|:-:|:-:|
| **BugFind-15** 🐛 | **80** | 60 | **+20** |
| **ToolCall-15** 🔧 | **97** | 93 | +4 |
| **DataExtract-15** 📄 | 81 | **83** | -2 |
| **InstructFollow-15** 📋 | **94** | 77 | **+17** |
| **ReasonMath-15** ➗ | 79 | 79 | 0 |
| **StructOutput-15** 📊 | **88** | 77 | **+11** |
| **Total** 🏆 | **86.5** | 78.2 | **+8.3** |

### Extended Evals

| Benchmark | Ours | Baseline | Notes |
|-----------|:----:|:--------:|-------|
| **GPQA-Diamond-198** 🔬 | **83.84%** | 73.7% | +10.14%, all 198 graded locally |
| **MMLU-500 (5-shot)** 📚 | **91.80%** | 91.6% | Approximately tied |

---

## 📦 Files

All GGUFs include YaRN 1M context scaling baked into the metadata — no extra flags needed.

| File | Size | Quant |
|------|:----:|:-----:|
| `...GGUF-Q2_K-YaRN-1M.gguf` | 11 GB | Q2_K (imatrix) |
| `...GGUF-Q3_K_M-YaRN-1M.gguf` | 13 GB | Q3_K_M (imatrix) |
| `...GGUF-Q4_K_M-YaRN-1M.gguf` | 16 GB | Q4_K_M (imatrix) |
| `...GGUF-Q5_K_M-YaRN-1M.gguf` | 19 GB | Q5_K_M (imatrix) |
| `...GGUF-Q6_K-YaRN-1M.gguf` | 21 GB | Q6_K (imatrix) |

### YaRN Parameters Embedded

| Parameter | Value |
|-----------|:-----:|
| `rope.scaling.type` | `yarn` |
| `rope.scaling.factor` | `4.0` |
| `rope.scaling.original_context_length` | `262144` |
| `rope.scaling.yarn_ext_factor` | `1.0` |
| `rope.scaling.yarn_attn_factor` | `1.0` |
| `rope.scaling.yarn_beta_fast` | `32.0` |
| `rope.scaling.yarn_beta_slow` | `1.0` |
| `context_length` | `1048576` |

---

## 🛠️ Usage

### Recommended Stack

```
🧩 OpenCode + LM Studio / llama.cpp
📐 Temperature: 0.6 · Top-p: 0.95
⚡ q4_k_m + mtp=3
🧠 Context: -c 1048576
```

### Quick Start (llama.cpp)

```bash
# Download (choose your quantization)
huggingface-cli download Brian6145/Qwen3.6-27B-Claude-Opus-DeepSeek-Distilled-Imatrix-MTP-1M-GGUF \
  <QUANT>-YaRN-1M.gguf --local-dir ./models

# Run with 1M context — YaRN params baked in
./llama-cli -m ./models/<QUANT>-YaRN-1M.gguf \
  --temp 0.6 --top-p 0.95 \
  -c 1048576 \
  -p "Your prompt here"

# Example with Q4_K_M:
./llama-cli -m ./models/Qwen3.6-27B-Claude-Opus-DeepSeek-Distilled-Imatrix-MTP-1M-GGUF-Q4_K_M-YaRN-1M.gguf \
  --temp 0.6 --top-p 0.95 \
  -c 1048576 \
  -p "Explain the entire codebase in this repository:"
```

> ⚠️ **1M context requires substantial memory.** At Q4_K_M, KV cache alone is ~15-20 GB. A100-80GB or equivalent recommended. For smaller GPUs, reduce `-c` to 128K or 256K.

### Via LM Studio

1. Load the GGUF file in LM Studio
2. Set backend to llama.cpp
3. Set context length to 1048576 (or your hardware limit)
4. Enable MTP (set depth=3) under inference options
5. Set `temperature = 0.6`, `top_p = 0.95`
6. Start the local API server
7. Connect via OpenCode or any OpenAI-compatible client

> 💡 **Pro tip**: For coding tasks, the `temp 0.6 / top_p 0.95` combo delivers the best balance of creativity and correctness.

---

## ⚠️ Known Limitations

### Evaluation Methodology
- **Opus-as-judge biases**: GLM5.2 comparisons are judge-evaluated, not absolute rankings
- **MMLU**: Single-run 5-shot result; variations in shot selection may cause fluctuation

### Capability Boundaries
- **DataExtract**: Scores 81 vs 83 baseline — extraction tasks may have slight regression from distillation
- **Closed-source teachers**: Risks include inherited biases and TOS compliance — assess for your use case
- **27B scale ceiling**: May still hit capacity limits on extremely complex long-chain reasoning

### Context Extension
- **YaRN is inference-only**: The 1M context window is achieved through RoPE scaling, not fine-tuning. Performance at extreme lengths may degrade gracefully rather than maintain full fidelity
- **Hardware requirements**: 1M KV cache at Q4_K_M requires ~15-20 GB VRAM just for the cache, plus ~16 GB for model weights. A100-80GB or multi-GPU setup recommended

### Deployment Notes
- **MTP=3**: Boosts throughput but adds VRAM overhead — disable or reduce on <24GB hardware
- **Imatrix**: Uses importance-matrix quantization, not standard k-quant — better parameter preservation at low bit widths
- **All quants included**: Unlike the base repo which only ships q4_k_m, this release includes Q2 through Q6 for flexibility

---

## 🔗 YaRN: Yet another RoPE extensioN Method

This release uses **YaRN** ([Peng et al., 2023](https://arxiv.org/abs/2305.13298)) to extend the native 262K context window to 1M tokens. YaRN is a state-of-the-art context extension method that:

- **Preserves existing capabilities** — no fine-tuning needed, the model retains all original knowledge
- **Uses NTK-aware scaling** — better allocation of frequency dimensions than linear interpolation
- **Adds a temperature ramp** — smooth transition between trained and extrapolated positions
- **Achieves strong perplexity** — outperforms linear and NTK-aware scaling at extended contexts

The parameters are baked directly into each GGUF file's metadata — llama.cpp reads them automatically at load time.

---

## 📝 Citation

```bibtex
@article{peng2023yarn,
  title = {YaRN: Efficient Context Window Extension of Large Language Models},
  author = {Bowen Peng and Jeffrey Quesnelle and Honglu Fan and Enrico Shippole},
  journal = {arXiv preprint arXiv:2305.13298},
  year = {2023}
}

@misc{opus-deepseek-distilled-q4m,
  title = {Opus-DeepSeek-Distilled-Q4M: A Distilled Agentic GGUF for Local Deployment},
  author = {Yin, Brian and BenchLocal Contributors},
  year = {2026},
  url = {https://github.com/brianyin/BenchLocal}
}
```

---

## 🙏 Acknowledgements

- **[Qwen](https://github.com/QwenLM/Qwen)** team — excellent foundational model
- **[Unsloth](https://github.com/unslothai/unsloth)** — efficient training infrastructure
- **[Merkyor](https://github.com/Merkyor)** — identified ReAct as the key to solving agent infinite loops
- **Community** — built on existing open-source exploration and practical experience

> *It is because of this work that came before that we can continue pushing forward, arriving at today's more stable, more practical, and more complete agent — and helping us get closer to the era of local agent AI.*