bowmanslayer/Qwen3.6-27B-Text-Only-W4A16-g128-ToolCalling

Qwen3.6-27B · Text-Only · W4A16-g128 · Tool-Calling

Run Qwen3.6-27B as a high-concurrency, long-context agent / tool-calling server on a single 24 GB GPU — vision stack removed, 4-bit quantized, FP8 KV-cache friendly.

This is a text-only, 4-bit weight (W4A16, group size 128, AutoRound) quantization of Qwen/Qwen3.6-27B. The multimodal vision encoder is fully removed and the weights are 4-bit quantized, bringing the model down to ~17 GB so a 24 GB consumer card (RTX 3090 / 4090) has room for a large KV-cache pool — the lever that actually decides multi-user concurrency and long-context capacity in production.

Community quantization. Not an official Qwen release; not endorsed by or affiliated with the Qwen team or Alibaba Cloud. "Qwen3.6" is used only to identify the upstream model this artifact derives from (Apache-2.0 §6).

🚀 Sibling release for low-latency single-stream: bowmanslayer/Qwen3.6-27B-Text-Only-W4A16-g128-MTP-ToolCalling — same weights with a BF16-preserved MTP head grafted in for speculative decoding (≈ +85% decode TPS, but trades away concurrency). Use that variant if your workload is single-user, decode-bound; use this variant if you need many concurrent users or long-context Q&A.

---

TL;DR — what you get (measured on 1× RTX 3090 24 GB, vLLM 0.20.2)

Mode	KV pool	Max context	Concurrency @ 16K	Decode TPS¹	When to use
Default (FP8 KV)	67,379 tokens	57,344 (56K)	~4×	~46 mean	Multi-user agent server / long Q&A ← recommended
FP16 KV (no FP8)	66,218 tokens	16,384	4.04×	~46 mean	Drop-in if you can't install CUDA toolkit

¹ Decode TPS reflects the 2026-05-22 Marlin metadata update. Pre-update baseline on the same hardware was 35.83 mean. Weights are byte-identical between update and pre-update — only config.json quantization metadata changed.

FP8 KV gives 3.5× larger max context at the same concurrency and the same decode TPS. On a 24 GB card this is the difference between a 16K-context server and a 56K-context server.

For higher single-stream decode speed (≈ +85% TPS, single user only), use the MTP sibling release.

---

What's new (2026-05-22) — Marlin kernel by default

The quantization_config metadata in both mm/ and self/ packagings was relabeled so vLLM auto-selects the mainline MarlinLinearKernel instead of the ConchLinearKernel Triton fallback. The weights themselves are byte-identical — only three metadata fields changed:

Field	Before	After
quantization_config.quant_method	auto-round	gptq
quantization_config.desc_act	(absent)	false
quantization_config.checkpoint_format	(absent)	gptq

This works because AutoRound was already configured with packing_format: auto_round:auto_gptq — the on-disk weight layout is GPTQ-compatible, so vLLM's gptq_marlin loader accepts the weights after relabeling.

Measured on 1× RTX 3090 24 GB, vLLM 0.20.2, FP8 KV + APC + --max-num-seqs 4:

	Before (Conch)	After (Marlin)	Delta
Decode TPS (5 prompts × 2 runs)	36.01 mean / 36.43 median	46.6 mean / 47.1 median	+29-30%
TTFT cold (3K system prompt)	2.94s	2.89s	-2% (GDN linear-attn prefill is the Ampere floor)
TTFT prefix-cache hit	n/a (APC off)	0.65s	(APC also enabled in update)
KV pool @ 16K	66,218 tok	104,086 tok (with FP8 KV+APC)	+57% from FP8 KV

The decode speedup is from MarlinLinearKernel accelerating the W4A16 GEMM (q/k/v/o_proj + MLP) — about half of the model's compute. The remaining half is the 48 Gated DeltaNet linear-attention layers (of 64 total), which still go through the Triton/FLA kernel and are the physical floor on prefill TTFT on Ampere SM86. Hopper+ benefits substantially less from this relabel because Marlin is engaged there even on the AutoRound metadata path.

To revert: each packaging ships config.json.bak-pre-marlin-20260522. Rename it back over config.json (no other files to touch) and you're back on the Conch path.

Runtime requirement change after this update:

• conch-triton-kernels is no longer required at runtime — Marlin is in vLLM mainline. Install only if you intend to revert.
• All other prerequisites (CUDA toolkit + ninja-build for FP8 KV) are unchanged.

---

Why this version exists

Qwen/Qwen3.6-27B ships only as full BF16 (52 GB) or FP8 (27 GB), with no official INT4, and the qwen3_5 hybrid architecture needs the very latest Transformers. None of these fit a single 24 GB GPU at any usable concurrency.

For self-hosted, on-prem agentic / chat workloads — multi-user, multi-turn, mixed short tool-calls and long Q&A — the binding constraint is KV-cache headroom:

• KV pool size × FP8 multiplier = how many concurrent users you can serve, or how long a single prompt+history can be.
• Vision tower weights cost ~4 GB of that headroom in the official model — dead weight for text-only agent pipelines.

This release strips the vision stack and prepares the model so vLLM's built-in FP8 KV cache path works end-to-end, doubling the practical KV pool on Ampere. The result is a 27B-class agent server that fits one 3090 with ~67K-token KV pool and 56K max context.

Who it's for: builders running on-prem / single-GPU agent and tool-calling pipelines on a 24 GB card, with multi-user concurrency or long Q&A context as the bottleneck. Who it's not for: anyone needing vision/multimodal (use the base model), or a full-precision quality baseline.

---

What was changed vs. the base model

A modified derivative of Qwen/Qwen3.6-27B. Per Apache-2.0 §4(b), modifications are stated prominently:

1. Vision encoder removed — text only. The base is a Causal LM with a Vision Encoder (modalities: text + vision (image & video); the base has no audio). Here the pure text backbone (Qwen3_5ForCausalLM) is extracted and quantized; the published weights contain zero vision/image/video tensors. Input and output are text only.
2. Weight-only quantization (AutoRound 0.12.3): bits=4, data_type=int, group_size=128, symmetric, packing auto_round:auto_gptq. "W4A16" (4-bit weights, 16-bit activations).
3. Kept at BF16 (not quantized) for quality: lm_head, model.embed_tokens, and every linear-attention layer's linear_attn.in_proj_a / in_proj_b.
4. Calibration: pile-val. A 512-line calibration pool was prepared; AutoRound was run with nsamples=256.
5. Tool-calling preserved — functionally validated.
6. Size: ~17 GB per packaging, vs. ~52 GB for the base BF16 weights.

No fine-tuning and no extra training data: a pure post-training quantization of the public base.

---

Two packagings in this repo

Both subfolders contain the same quantized weights (1651 tensors each); only the wrapper layout and key namespacing differ.

Subfolder	architectures	Use it when
self/	Qwen3_5ForCausalLM	Generic Transformers / custom loaders
mm/	Qwen3_5ForConditionalGeneration + language_model_only: true	vLLM (recommended) — this is the packaging served in production

mm/ exists because vLLM registers Qwen3.6 under Qwen3_5ForConditionalGeneration; it wears that multimodal "shape" while running purely as a language model (language_model_only: true). Neither packaging contains any vision weights.

---

Hardware & runtime

• Single NVIDIA 24 GB GPU. Validated on RTX 3090 (Ampere, SM86).
• Ampere kernel: as of the 2026-05-22 update, vLLM auto-selects mainline MarlinLinearKernel via the relabeled gptq metadata path. conch-triton-kernels is no longer required at runtime; install only if you intend to revert the metadata to test against Conch. On Hopper+ mainline kernels apply throughout.
• For the FP8 KV path (recommended):
  • CUDA toolkit (nvcc 13.x) installed and on CUDA_HOME — flashinfer JIT-compiles fp8 attention kernels at runtime.
  • ninja-build system package.
  • Then --kv-cache-dtype fp8 works on this model. Without these, fall back to default fp16 KV.
• Versions: validated on vLLM 0.20.2. transformers >= 5.8.1 required for the qwen3_5 architecture.
• Served with --dtype float16.

Measured KV-cache footprint (vLLM startup profiler, 1× RTX 3090 24 GB)

--gpu-memory-utilization 0.93 --dtype float16 --language-model-only. Numbers verbatim from vLLM's startup KV profiler (warm torch.compile cache where applicable):

--max-model-len	KV dtype	Available KV cache memory	GPU KV cache size	Maximum concurrency @ max-len
16,384	fp16 (default)	4.65 GiB	66,218 tok	4.04×
16,384	fp8	4.30 GiB	104,155 tok	6.36×
32,768	fp8	2.25 GiB	64,170 tok	1.96×
49,152	fp8	2.25 GiB	66,004 tok	1.34×
57,344 (production)	fp8	2.23 GiB	65,945 tok	1.15×
65,536	fp8	—	—	❌ GDN warmup CUDA error (Ampere edge case at large T)

Tested upper bound on this hardware: 57,344 tokens at fp8 KV. Going further triggers a Gated-DeltaNet warmup kernel illegal-memory-access at boot time. If you hit that, lower --max-model-len.

---

Usage — vLLM (use the mm/ packaging)

Recommended: FP8 KV at 56K max context

# Prereqs (one-time):
#   1) conch-triton-kernels installed in your vllm venv (Ampere W4A16 g128 kernel)
#   2) CUDA toolkit 13.x installed; CUDA_HOME and PATH point at it
#   3) ninja-build system package

CUDA_HOME=/usr/local/cuda-13.0 \
PATH=/usr/local/cuda-13.0/bin:$PATH \
vllm serve bowmanslayer/Qwen3.6-27B-Text-Only-W4A16-g128-ToolCalling \
  --tokenizer bowmanslayer/Qwen3.6-27B-Text-Only-W4A16-g128-ToolCalling \
  --served-model-name qwen3.6-27b \
  --language-model-only \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --gpu-memory-utilization 0.93 \
  --max-model-len 57344 --max-num-seqs 8 \
  --dtype float16 --kv-cache-dtype fp8
# point vLLM at the mm/ subfolder

This gives a ~65K-token KV pool that can serve:

• 1 request at the full 56K context (long Q&A / RAG), or
• 4 requests at 16K each (typical agent conversations), or
• 8 requests at 8K each (short tool-calls).

Fallback: FP16 KV at 16K (no CUDA toolkit needed)

vllm serve bowmanslayer/Qwen3.6-27B-Text-Only-W4A16-g128-ToolCalling \
  --language-model-only \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder --reasoning-parser qwen3 \
  --gpu-memory-utilization 0.93 \
  --max-model-len 16384 --max-num-seqs 8 \
  --dtype float16

Same TPS, same concurrency at 16K, but capped at 16K max context.

Transformers (use the self/ packaging)

Requires transformers >= 5.8.1 (for the qwen3_5 architecture) and an AutoRound/AutoGPTQ-aware loader. Load from the self/ subfolder.

Qwen3.6 enables a "thinking" mode by default which consumes extra tokens. For tool dispatch you may want to disable it (enable_thinking=false) to reduce token usage and latency.

---

For higher single-stream decode speed

This variant is optimized for concurrency + context, not single-stream latency. Its decode TPS is ~36 — the same as a vanilla 27B int4 W4A16 on this hardware.

If you have a single-user, decode-bound workload (e.g. interactive chat where each user wants their tokens streamed as fast as possible) the MTP sibling release is the right pick:

• bowmanslayer/Qwen3.6-27B-Text-Only-W4A16-g128-MTP-ToolCalling
• Same quantized weights + a BF16-preserved Multi-Token Prediction head grafted in
• Decode TPS ≈ 66 (n=3) or 54 (n=1) — +85% / +50% vs this variant
• Cost: KV pool shrinks to ~16-32K, concurrency drops to 1-2× — single-user only

You can also combine the two: run the MTP variant with --kv-cache-dtype fp8 and --speculative-config '{"method":"mtp","num_speculative_tokens":1}' for a middle ground (KV ≈ 33K, ~2× concurrency, decode TPS ≈ 47).

For long-context tuning (60K+) consider community projects like Sandermage's Genesis vLLM patches — note we observed that the upstream Genesis 3090-1x-tool-agent preset OOMs with this specific text-only-stripped model layout on a 24 GB 3090 (preallocs ~22 GB regardless of preset config), so it's better suited to the vision-preserved upstream variants today.

---

Validation

Functional validation only — not a full benchmark. Under vLLM 0.20.2:

• Tool-calling: 24/24 on internal quick-test set
• Lightweight reasoning: 10/10
• FP8 KV path: measured the KV-cache table above on a single RTX 3090 with vLLM's startup profiler. Bench TPS at 46.6 mean / 47.1 median with the 2026-05-22 Marlin update (5 prompts × 2 runs, --max-model-len 16384 --max-num-seqs 4 --kv-cache-dtype fp8 --enable-prefix-caching --gpu-memory-utilization 0.97). Pre-Marlin baseline on the same hardware was 36.01 mean / 36.43 median at --max-model-len 57344 --max-num-seqs 4. The decode speedup is primarily attributable to Marlin; FP8 KV and APC mainly affect KV-pool size and prefix-cache TTFT.

Not a full lm-eval comparison against the BF16 base. Quantization can change outputs vs. the base model.

---

License & attribution

• Base model: Qwen/Qwen3.6-27B, © 2026 Alibaba Cloud, Apache License 2.0.
• This artifact is a modified derivative (vision removed; weight-only W4A16 quantization), distributed under the same Apache-2.0 license. Verbatim license copy: LICENSE (§4(a)); modifications listed under What was changed (§4(b)); upstream copyright/attribution retained (§4(c)); see also NOTICE.
• Trademark (§6): "Qwen" / "Qwen3.6" are used only to identify the upstream model. This release is not official and implies no endorsement or affiliation.
• Provided "AS IS" (Apache-2.0 §7); quantization may change outputs vs. the base model.

Citation

Upstream model:

@misc{qwen3.6-27b,
    title  = {{Qwen3.6-27B}: Flagship-Level Coding in a {27B} Dense Model},
    author = {{Qwen Team}},
    month  = {April},
    year   = {2026},
    url    = {https://qwen.ai/blog?id=qwen3.6-27b}
}

Acknowledgments · 致谢

This release is dedicated to my wife. When I worried about the business failing, she told me she'd happily support me at home so I could keep working on AI.

妻如此，夫复何求。

Released on 5月20日 · 520 — with love.