Qwen3.6-27B — PrismaQuant gridbook 5.5-bit (gridbook formats, vLLM)

23.0 GB codebook-quantized Qwen3.6-27B (vision-language, MTP included) serving through stock vLLM with the open gridbook plugin — no fork, no core patches. Weights are stored as product-VQ codebook indices whose codebook values lie exactly on hardware grids (e4m3 / E2M1), so serving decodes into bit-standard FP8/NVFP4 tensors and runs on the same tensor-core GEMM paths a conventional checkpoint uses.

Measured quality (held-out KL vs the BF16 teacher)

Protocol: top-20 prompt_logprobs KL on a 12.3k-token held-out corpus (disjoint from calibration), 8,176 positions, all arms served back-to-back on one box against the same teacher capture — a strict relative comparison.

artifact	size	KL (all)	KL (confident half)	top-1 (conf)
this artifact (CB ladder, 5.5 bpp body)	23.0 GB	0.0049	0.00295	99.88%
Qwen3.6-27B-PrismaAURA-5.5bit (NVFP4+FP8)	23 GB	0.0211	0.0130	99.58%
Qwen3.6-27B-PrismaSCOUT 5p31	20.6 GB	0.0344	0.0249	99.04%

At matched size and matched body bitrate, the codebook ladder carries −77% KL versus the strongest conventional (NVFP4+FP8) artifact in this family. Same model, same bytes, same protocol — the delta is the format.

Pre-ship validator: PASS — mean NLL 1.400/tok, p99 NLL 1.807 (thresholds 3.0 / 6.0)
ToolEvalBench (hardmode, seed 1234, sequential): 87 / 100 (129/148) — family anchors on identical protocol: BF16 teacher 86, PrismaSCOUT 85, PrismaAURA-5.5 91; repeat churn band ±2–3

What's inside

section	format	notes
504 body Linears	FP8-CB K36–K47 (426 units) + BF16 (69)	per-Linear rungs chosen by measured error under a byte budget; K47 (5.88 bpw) ×175, K44 ×109, K40 ×108 — the ⅛-bpw all-integer ladder in actual use
vision tower (110 Linears)	NVFP4 weight-only (W4A16)	quantized ViT, verified on image inputs
MTP draft block	BF16	canon throughput selector: highest-fidelity rung that fits
embeddings + lm_head	BF16	248k vocab, untied

Full per-Linear assignment ships in allocation/. Codebooks (cb_codebooks.pqcb) are shared per role family and total a few MB.

Serve this model


Plugin	`gridbook` — an out-of-tree vLLM quantization plugin. Stock vLLM, no fork, no core patches.
GPU	NVIDIA Blackwell, compute capability `sm_120` / `sm_121`. Measured on GB10 / DGX Spark (`sm_121`). On older GPUs the plugin still loads but runs its Triton fallback kernels — correct, not fast, and not a production serving target.
Memory	20.8 GiB weights + KV cache, measured on one 128 GB unified-memory GB10 (`Model loading took 20.82 GiB`). Before gridbook 0.1.0 the same artifact loaded at 35.86 GiB — dense codebook weights were resident twice (issue #1, fixed in 0.1.0); use 0.1.0 or newer. A 32 GB consumer Blackwell (RTX 5090, `sm_120`) should now fit weights plus a useful KV budget, and the issue reporter measured 20.54 GiB with 32k context using equivalent patches — but we have not verified 0.1.0 on a 32 GB card ourselves.
vLLM	Served here on the `vllm-node` image used for the 0.1.0 verification; also exercised on 0.25.1 and 0.23.1rc1.dev1060 by an external reporter (issue #1). The plugin surface is small but vLLM internals drift — pin a version you have tested.
Toolchain	CUDA toolkit with `nvcc` on `PATH` in the serving container — the plugin JIT-builds its kernels on first model load (~30 s, cached). `nvcc` 13.0 is the tested toolchain.
Parallelism	Single GPU (`tp=1`). Tensor parallelism is not implemented in the plugin.

pip install gridbook   # JIT-builds kernels for your GPU (capability >= 8.0 floor)
vllm serve rdtand/Qwen3.6-27B-prismaquant-gridbook-5.5bit-vllm \
  --max-model-len 32768 --gpu-memory-utilization 0.90

Honest limitations

KL numbers are the relative protocol above, not the full-vocab exact-KL contract; the ranking is protocol-internal and same-session.
ToolEvalBench ordering vs PrismaAURA-5.5 may differ from the KL ordering (task metrics churn ±2–3 points between serving configs).
One box, one seed per benchmark.

Quantized with PrismaQuant (measurement-driven per-Linear format allocation); format spec and kernels: gridbook, Apache 2.0.

Contact: robert.tand@icloud.com