XpressAI/Qwen3.6-35B-A3B-ExpertClone-Logic-UD-Q4_K_XL-GGUF

Qwen3.6-35B-A3B — Specialized Expert Cloning (Logic)

A modified version of Qwen3.6-35B-A3B produced by Specialized Expert Cloning (SEC) — a new MoE-native weight-surgery technique. No training, no added parameters, no change to the model's architecture or layer count. Router weights for a set of low-usage experts are replaced with copies of logic-specialized experts' weights, so the router has multiple slots that compute the same function for logic-relevant tokens.

---

Files

File	Layers	Experts	Active / Token	Size
Qwen3.6-35B-A3B-expert_clone_logic-UD-Q4_K_XL.gguf	40	256	8	20.8 GiB

The base GGUF (no surgery) lives at unsloth/Qwen3.6-35B-A3B-GGUF.

---

Probe scores

Scores from the internal sweep probe. Sample sizes are small — directional, not definitive.

Metric	Base	Cloned	Δ
Math (GSM8K-style partial credit)	0.625	0.563	−0.06
EQ (EQ-Bench-style, 0–100)	83.5	94.9	+11.4
Reasoning total (17 probes, 5 categories)	0.765	0.882	+0.12
↳ causal	0.00	0.67	+0.67
↳ date, logic, navigation, GSM	all unchanged

The cloned model recovers a previously-broken reasoning subcategory (causal, 0/3 → 2/3), gains ~11 points of EQ on our probe, and loses ~6 points of math. We publish it as a working demonstration of the technique, not as a claim of strictly-better performance across all benchmarks.

---

What's different about MoE

Qwen3.6-35B-A3B is a hybrid Mamba/attention MoE:

• 40 layers, full_attention_interval = 4
• 256 experts per layer, top-8 routing per token
• One shared expert per layer, always active
• ~35B total params, ~3B active per token

Conventional layer-level surgery (RYS / layer duplication) works on MoE mechanically but provides only marginal gains here — the base model is already strong on EQ and reasoning, leaving little room to exploit. Instead, we target the router.

---

How the clone was made

Step 1 — Expert profiling. Using llama-debug --tensor-filter ffn_moe_topk-\d+ we logged which experts the router fires on 24 short prompts spanning six domains (math, code, logic, EQ, factual, creative). Per (layer, expert) we computed the firing rate for each domain and the cross-domain mean rate.

Step 2 — Donor selection. For each of the 40 layers, we picked the single expert with the highest (logic-rate ÷ mean-rate) ratio. Many hit the natural ceiling (6×, meaning the expert fires for logic and essentially nothing else) — evidence that experts specialize strongly during training.

Step 3 — Victim selection. For each layer, we picked the expert with the lowest total firing count across all six domains — i.e., the expert the router almost never chose for any of our probes. These are the "junk" slots we can overwrite without losing capability the model was using.

Step 4 — Byte-level surgery. In each of the four per-layer MoE tensors (ffn_gate_inp.weight, ffn_gate_exps.weight, ffn_up_exps.weight, ffn_down_exps.weight), we copied the donor expert's per-expert slab into the victim's slot. Because the expert index is the outermost storage axis, this is a fixed-size byte copy that works uniformly for F32 router weights and for Q4_K / Q5_K quantized expert weights.

Total: 40 donor→victim pairs (one per layer) × 4 tensors = 160 slab copies. No training, no extra parameters, same file size as the base GGUF.

---

Why this works (hypothesis)

The router computes a softmax over 256 expert logits and routes each token to its top-8 experts. Because a donor and its victim now share identical router columns and identical expert weights, they produce identical logits for any token. So:

1. When the donor already scored in the top-8 for a logic-ish token, the victim scores identically and also enters the top-8, displacing what would have been the 8th-ranked expert.
2. The gated sum now weights the donor's computation roughly twice as much as it would have, because two of the eight selected slots contain it.
3. For tokens the donor was not relevant to, it didn't score high before and still doesn't, so the victim slot also doesn't activate — no harm done.

Net effect: on-distribution for logic-ish tokens, the donor's contribution is amplified; off-distribution, behavior is nearly unchanged.

---

Usage

llama.cpp / llama-server

llama-server -m Qwen3.6-35B-A3B-expert_clone_logic-UD-Q4_K_XL.gguf \
             -ngl 99 --port 8080

Thinking mode

Qwen3.6 defaults to thinking mode (<think>…</think>). Add /no_think to the system prompt for fast direct answers.

VRAM

~25 GiB on an A100 with 4k context and Q8 KV cache.

---

Reproducibility

Tooling and probe prompts are in the llm-circuit-finder repo:

• moe_profiler/probes.py — the 24 probe prompts
• moe_profiler/profile.py — runs llama-debug on each prompt and parses routing
• moe_profiler/analyze.py — computes specialization ratios
• expert_clone.py — performs the GGUF tensor surgery

---

Limitations

• Small probe. Results come from 24 short prompts and a 17-question reasoning suite. Confirming on larger benchmarks (MMLU, BBH, GSM8K, HumanEval, EQ-Bench, TruthfulQA) is needed before drawing strong conclusions.
• Domain-targeted improvement. We targeted logic and observed gains in causal reasoning and EQ. Math regressed slightly. Other subdomains may regress more on tests we did not run.
• Victim selection is a coarse proxy. We treat "lowest total firing across our 6 probe domains" as "unused by the model." A more principled approach (e.g., profiling on a larger corpus, or picking victims whose removal minimally perturbs a small validation perplexity) would likely select better victims.

Credits

• David Ng for RYS, which motivated this work
• Unsloth for the base GGUF quantization
• Qwen team for Qwen3.6-35B-A3B
• llama.cpp for local inference and the tensor-introspection hooks used to profile routing

License

Apache 2.0 (inherited from base model).