--- license: apache-2.0 base_model: Qwen/Qwen3.6-27B-Instruct tags: - gguf - qwen3.6 - rys - layer-surgery - reasoning - bfcl - function-calling - speculative-decoding - dflash language: - en --- # Qwen3.6-27B — RYS Layer Surgery (GGUF) A modified version of [Qwen3.6-27B-Instruct](https://huggingface.co/Qwen/Qwen3.6-27B-Instruct) produced by **RYS layer duplication** — no training, no weight changes, just running layers 33–36 a second time during the forward pass. Based on [David Ng's RYS method](https://dnhkng.github.io/posts/rys/). --- ## TL;DR On the Berkeley Function-Call Leaderboard (BFCL v4, 100 tests/category × 13 single-turn categories, sampled), this variant **beats the unmodified base model by +1.96 pp on average** when run with thinking mode enabled — driven by large gains on the hardest live categories: | Category | Base | rys_33-36 | Δ | |---|---|---|---| | live_parallel | 68.75% | **87.50%** | **+18.75** | | live_relevance | 68.75% | **81.25%** | **+12.50** | | live_parallel_multiple | 70.83% | **75.00%** | **+4.17** | | **mean (13 categories)** | **82.56%** | **84.52%** | **+1.96** | The wins come from improved reasoning during prefill on multi-call / relevance-judgement queries. The trade is small regressions (−1 to −3 pp) on easier non-live categories. **Thinking mode is required** — without it, this variant slightly underperforms base. --- ## Files | File | Quant | Layers | Size | Role | |------|-------|--------|------|------| | `Qwen3.6-27B-rys_33-36-UD-Q4_K_XL.gguf` | Q4_K_XL | 68 | 18 GiB | target model | | `Qwen3.6-27B-rys_33-36-Q8_0.gguf` | Q8_0 | 68 | 29 GiB | target model | | `Qwen3.6-27B-DFlash-Q8_0-rys.gguf` | Q8_0 | 5 | 1.8 GiB | DFlash draft (speculative decoding) | The base GGUF (no surgery) is at [unsloth/Qwen3.6-27B-GGUF](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF). --- ## Internal probe results A small probe of math, EQ, and reasoning prompts was run during the layer search. The probe categories are tiny (3 questions per reasoning subcategory, ~16 EQ-Bench-style items, ~16 math problems) so individual numbers should be treated as directional, not definitive. | Metric | Base | rys_33-36 | |---|---|---| | Math (GSM8K-style partial credit) | 0.537 | 0.500 | | EQ (EQ-Bench-style, 0–100) | 93.59 | 86.64 | | Reasoning total (17 probes, 5 subcategories) | 0.765 | **0.882** | |   ↳ causal | 0.67 | **1.00** | |   ↳ date | 1.00 | 1.00 | |   ↳ logic | 1.00 | 1.00 | |   ↳ navigation | 0.67 | **1.00** | |   ↳ gsm | 0.60 | 0.60 | Layers 33–36 was the only configuration in the layer-block sweep that achieved a perfect score on the *causal* reasoning subcategory while keeping the other reasoning categories at or above their baseline. This is what motivated picking it for the BFCL run below. --- ## BFCL results (sampled, thinking enabled) | Category | Base | rys_33-36 | |---|---|---| | irrelevance | 90.00 | 88.00 | | multiple | 96.00 | 95.00 | | parallel | 93.00 | 91.00 | | parallel_multiple | 87.00 | 85.00 | | simple_java | 59.00 | 61.00 | | simple_javascript | 74.00 | 72.00 | | simple_python | 95.00 | 92.00 | | live_irrelevance | 98.00 | 99.00 | | live_multiple | 88.00 | 87.00 | | live_parallel | 68.75 | **87.50** | | live_parallel_multiple | 70.83 | **75.00** | | live_relevance | 68.75 | **81.25** | | live_simple | 85.00 | 85.00 | | **mean** | **82.56** | **84.52** | Sample size: 100 tests/category for categories with ≥100 entries; the full category was used for the smaller ones (`live_parallel`, `live_parallel_multiple`, `live_relevance`, `simple_javascript`). 1006 tests per model in total. The full benchmark would be ~5x larger and would also cover multi-turn, memory, and web-search categories that we did not run. Inference: llama.cpp `llama-server --jinja`, BFCL via `/v1/chat/completions` with native tool use, `temperature=1.0, top_p=0.95, top_k=20, max_tokens=8192`. Multi-turn, memory, and web-search categories were not run. --- ## What is RYS? Transformers self-organise during training into functional **circuits** — contiguous blocks of layers that act together. RYS duplicates a specific block in the forward pass using the same weights: ``` Normal: 0 → … → 32 → 33 → 34 → 35 → 36 → 37 → … → 63 rys_33-36: 0 → … → 32 → 33 → 34 → 35 → 36 → 33 → 34 → 35 → 36 → 37 → … → 63 ``` The model processes layers 33–36 twice. No fine-tuning, no extra parameters beyond the GGUF file overhead. Total layer count goes from 64 → 68. --- ## How the layer range was found A two-pass sweep across all 64 layers using a small probe of math, EQ, and reasoning prompts: - **Pass 1** (8-layer blocks, stride 4): identified hot zones around layers 32–48 (math gains, causal reasoning) and 48–60 (general reasoning gains). - **Pass 2** (4-layer blocks, stride 1, layers 32–58): `(33, 37)` was the only configuration that achieved a perfect score on the probe's *causal* reasoning subcategory while keeping `date`, `logic`, and `nav` at their baseline ceilings. The probe alone suggested rys_33-36 was a moderate win. The **sampled BFCL run with thinking enabled confirms it on the harder live categories** (above). ### Extended evaluation (Ng's protocol) After a thoughtful question on the discussion forum about deviations from [David Ng's suggested reproduction path](https://github.com/dnhkng/RYS#suggested-reproduction-path), we went back and ran the steps we had skipped: **Extended probe** — `math_120 + eq_140` from Ng's repo, `--reasoning off` to match the protocol's intent (the math probe is designed for intuitive guessing, not deliberate computation): | Variant | math_120 | eq_140 | |---|---|---| | base | 0.9986 | 74.53 | | **rys_33-36** | 0.9930 | **78.81** | On the larger probe rys_33-36 holds its EQ improvement (+4.28 pp). Math is at ceiling for both. Note this is the *opposite* direction from our small internal probe (where rys_33-36 had lower EQ) — small-probe variance was misleading us; the 140-question sample is the trustworthy reading. **Depth-2 beam search** — 10 non-overlapping pair-combinations of the top single-block configs, each scored on the same probe: | Variant | math_120 | eq_140 | |---|---|---| | **rys_33-36** | 0.9930 | **78.81** | | rys_33-36 + 49-52 | 0.9226 | 75.66 | | rys_33-36 + 53-56 | 0.9219 | 75.27 | | rys_33-36 + 54-57 | 0.9639 | 72.21 | | rys_33-36 + 56-59 | 0.9643 | 74.21 | | rys_33-36 + 58-61 | 0.9930 | 68.78 | | rys_49-52 + 53-56 | 0.8864 | 66.70 | | rys_49-52 + 56-59 | 0.9654 | 69.67 | | rys_49-52 + 58-61 | 0.9606 | 69.18 | | rys_53-56 + 58-61 | 0.9635 | 63.57 | | rys_54-57 + 58-61 | 0.9703 | 59.93 | No depth-2 combination beats rys_33-36 on EQ_140. Stacking blocks degrades math (sometimes catastrophically) without improving EQ. So the shortcut we took in candidate selection (no beam search) did not cost us a better configuration in this neighborhood. We did **not** train Ng's surrogate regressor or run a deeper beam search — those would explore more of the configuration space and might find something better. --- ## Hybrid Mamba/attention architecture constraint Qwen3.6-27B is a **hybrid SSM/attention model** (`full_attention_interval = 4`): full attention every 4th layer, Gated DeltaNet SSM everywhere else. This creates a hard constraint: **the total layer count must remain divisible by 4**. - Block size 4 → 64 + 4 = 68 layers (68 ÷ 4 = 17 ✓) - Block size 3 → 64 + 3 = 67 layers (67 ÷ 4 = 16.75 ✗ → crash) --- ## Usage ### llama.cpp / llama-server The wins **require thinking mode**. Use `--jinja` so the server applies the Qwen3.6 chat template, which primes thinking properly: ```bash llama-server -m Qwen3.6-27B-rys_33-36-UD-Q4_K_XL.gguf \ --jinja \ -ngl 99 -c 32768 \ --port 8080 ``` ### Sampling parameters (Qwen3.6 thinking-mode defaults) ``` temperature = 1.0 top_p = 0.95 top_k = 20 min_p = 0.0 ``` For more deterministic / coding-focused tasks, Qwen recommends `temperature=0.6` instead. Either way, leave thinking enabled. ### Token budget Qwen3.6's thinking chains can be long (we observed up to ~7k tokens of reasoning on hard BFCL parallel cases). Set `max_tokens` ≥ 8192 to avoid truncating mid-thought. ### VRAM About 22 GiB at Q4_K_XL with 32k context and Q8 KV cache. Fits comfortably on a single A100 40 GB. ### Speculative decoding with DFlash (faster inference) RYS makes the model deeper (68 vs 64 layers), so each token costs a little more compute. You can win that back — and then some — with **DFlash speculative decoding**, a diffusion-style draft head that proposes a block of tokens per step which the target model verifies in one pass. `Qwen3.6-27B-DFlash-Q8_0-rys.gguf` is a DFlash draft **trained against the 68-layer RYS layout** — its hidden-state taps (`target_layer_ids = [1, 16, 31, 50, 65]`) index into the duplicated-block arrangement, so it must be paired with the RYS target, not the stock base model. DFlash support is not in upstream llama.cpp yet; use [BeeLlama](https://github.com/Anbeeld/beellama.cpp), a llama.cpp fork that adds the `dflash` spec type: ```bash ./build/bin/llama-server \ -m ~/models/Qwen3.6-27B-rys_33-36-UD-Q4_K_XL.gguf \ --spec-draft-model ~/models/Qwen3.6-27B-DFlash-Q8_0-rys.gguf \ --spec-type copyspec,dflash \ --spec-dflash-cross-ctx 1024 \ --port 9999 -np 1 --kv-unified \ -ngl all --spec-draft-ngl all \ -b 2048 -ub 1024 --flash-attn on \ --jinja --no-host --reasoning on \ --chat-template-kwargs '{"preserve_thinking":true}' \ --temp 0.6 --top-k 20 --top-p 1.0 --min-p 0.0 ``` The draft adds ~1.8 GiB of VRAM. Acceptance is highest on the long, structured thinking chains this model produces, which is exactly where the RYS variant spends its tokens. --- ## When to use this - You want better function-calling performance on **complex live queries** (parallel calls, relevance judgement) and you can afford ~6 extra layers of prefill compute. - You're running with **thinking mode on** (this is where the gain comes from). ## When NOT to use this - You're running **without thinking** — base will be ~1.5 pp better. - You care about the very-easy categories (`simple_python`, `multiple`) more than the hard live ones — base is 1–3 pp better there. --- ## Credits - [David Ng](https://dnhkng.github.io/posts/rys/) for the original RYS method - [Unsloth](https://huggingface.co/unsloth) for the base `UD-Q4_K_XL` quantization - [Qwen team](https://huggingface.co/Qwen) for Qwen3.6-27B - [llama.cpp](https://github.com/ggml-org/llama.cpp) for local inference - The [Berkeley Function-Call Leaderboard](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard) for the eval harness ## License Apache 2.0 (inherited from base model).