thetom-ai/Qwen3.6-35B-A3B-ConfigI-GGUF

Qwen3.6-35B-A3B — TurboQuant+ Config-I (GGUF, experimental)

Experimental Config-I quantization of Qwen/Qwen3.6-35B-A3B for llama.cpp, produced with the feature/turboquant-kv-cache branch.

Status: experimental. Metal coherency looks clean across contexts (see samples below). CUDA has not been tested yet — if you run this on a CUDA build of the fork, please open an issue with your results. Use at your own risk.

Details

Source	Qwen/Qwen3.6-35B-A3B (BF16)
Quant type	TQ3_1S (4.0 bpw WHT-rotated, per-tensor)
Output size	~16.4 GB
Architecture	qwen35moe (GatedDeltaNet hybrid + 256-expert MoE)
Intended runtime	TheTom/llama.cpp, branch feature/turboquant-kv-cache

Runtime

This GGUF is intended for TheTom's llama.cpp fork. Upstream llama.cpp does not (yet) support the TQ3_1S weight type or the TurboQuant asymmetric KV path, so loading/running will likely fail outside the fork.

git clone https://github.com/TheTom/llama.cpp
cd llama.cpp
git checkout feature/turboquant-kv-cache
cmake -B build -DGGML_METAL=ON && cmake --build build -j
build/bin/llama-bench -m Qwen3.6-35B-A3B-ConfigI.gguf -p 128 -n 32 -ngl 99

Coherency (Metal, feature/turboquant-kv-cache)

Sanity generations on an M5 Max 128GB using the production KV recipe (-ctk q8_0 -ctv turbo3 -fa 1). Prompt is the opening line of The Great Gatsby with "Summarize the passage:" appended. Reasoning traces preserved; responses trimmed for brevity. The model correctly identifies Gatsby at every context length and flags the missing passage body.

ctx ≈ 128

<think>
1. Analyze User Input:
   - Quote: "In my younger and more vulnerable years my father gave me some advice."
   - Request: "Summarize the passage:"
   - Key Observation: This is the famous opening line of F. Scott Fitzgerald's
     The Great Gatsby. The user is asking to "summarize the passage," but only
     provided a single sentence.
</think>

ctx ≈ 1024

The phrase you provided is the opening line of F. Scott Fitzgerald's novel
*The Great Gatsby*.

Since you only provided the opening sentence, here is a summary of its
significance and context:

Summary of the Passage (Opening Line)
> "In my younger and more vulnerable years my father gave me some advice."

This sentence establishes the narrator's reflective tone …

ctx ≈ 4096

<think>
1. Analyze User Input:
   - The user quotes the famous opening line: "In my younger and more
     vulnerable years my father gave me some advice."
   - They ask: "Summarize the passage:"
   - Wait, there's no passage provided after the colon. The user only
     provided the opening line and then asked for a summary of "the passage,"
     but didn't include the passage body.
</think>

ctx ≈ 16384

<think>
1. Analyze User Input:
   - The user provides a famous opening line: "In my younger and more
     vulnerable years my father gave me some advice."
   - They ask to "Summarize the passage:"
   - But there's no passage provided! The prompt is incomplete.
2. Identify Key Issue:
   - The user quotes the opening line of F. Scott Fitzgerald's The Great …
</think>

ctx ≈ 32768

The passage "In my younger and more vulnerable years my father gave me some
advice" is the famous opening line of F. Scott Fitzgerald's novel
*The Great Gatsby*.

It sets the tone for the entire book, which is a first-person retrospective
narrative. The narrator, Nick Carraway, looks back on his experiences in
Long Island and New York during the summer of …

Metal verdict. Coherent and on-topic at every context length tested, including the 32k full-context run. No drift, no repetition, no nonsense. Quality looks good at this compression level based on coherence alone — a full PPL/MMLU sweep has not been run.

CUDA: not tested yet. If you run this on a CUDA build of the fork, please open an issue on the fork with your results.

Links

• Config-I paper
• TurboQuant+ repository
• MLX counterpart (same policy, standard MLX format): thetom-ai/Qwen3.6-35B-A3B-ConfigI-MLX

---

Quantized by @thetom-ai · GitHub · X · Sponsor