Hugging Face's logo

rdtand
/
Qwen3.6-27B-PrismaAURA-5.5bit-vllm

Safetensors
vllm
qwen3_5
compressed-tensors
nvfp4
fp8
speculative-decoding
prismaquant
prismaaura
aura
mixed-precision
blackwell
Model card Files
xet
 Community
1
Qwen3.6-27B-PrismaAURA-5.5bit-vllm
File size: 4,466 Bytes
d017bf9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
---
license: apache-2.0
base_model: Qwen/Qwen3.6-27B
library_name: vllm
tags:
  - qwen3_5
  - compressed-tensors
  - nvfp4
  - fp8
  - vllm
  - speculative-decoding
  - prismaquant
  - prismaaura
  - aura
  - mixed-precision
  - blackwell
---

# Qwen3.6-27B — PrismaAURA 5.5 (Blackwell, NVFP4 + FP8 + BF16)

PrismaQuant export of `Qwen/Qwen3.6-27B` for vLLM `compressed-tensors` serving on
NVIDIA Blackwell. This is the **PrismaAURA** artifact: a per-Linear mixed-precision
allocation chosen by **AURA** — PrismaQuant's production-faithful KL–Fisher
allocator — over the full `NVFP4 / FP8 / BF16` format menu.

Where the earlier PrismaSCOUT artifact used only NVFP4 + BF16, PrismaAURA adds the
FP8 middle rung to the menu, so the allocator can spend an extra ~4 bits on the
Linears that need it instead of jumping all the way to 16. ~5.5 bits per
quantizable parameter; **23 GB** on disk.

## Tool-use fidelity (ToolEvalBench, hardmode, deterministic)

Tool-use is the metric we weight most: a small probability shift at a decision
point can flip a tool call. On ToolEvalBench (`--no-think --hardmode`, sequential,
`temperature=0`, `seed=1234`), PrismaAURA scores the **highest of the entire
family — above full precision**:

| Artifact | ToolEvalBench |
|---|---:|
| **Qwen3.6-27B PrismaAURA 5.5 (this)** | **91 / 100** (134/148) |
| Qwen3.6-27B PrismaSCOUT 5.31 (prior flagship) | 85 / 100 |
| Qwen3.6-27B BF16 (full precision) | 86 / 100 |

Same harness, same seed for all three. PrismaAURA preserves tool-calling behavior
at least as well as the unquantized model on this benchmark, at 5.5 bpp.

## Served KL-vs-BF16

KL divergence measures how far the quantized model's full output distribution has
drifted from the original full-precision model (0 = identical). Measured on a held
WikiText split (exact vLLM, n=8 × seqlen 512, vs the BF16 teacher in the same
session):

- **Served KL-vs-BF16: 0.0342**

Against the prior AURA research build at the same bpp (NVFP4+BF16 only, earlier
render/export code), this is a **−40.9% reduction** in served KL — driven by the
full FP8 menu, a corrected per-Linear render (fixed GPTQ damping, scale-faithful
NVFP4 export), and a corrected calibration probe. (Single-draw served KL; the
direction is corroborated by the deterministic ToolEvalBench result above.)

## What AURA does

A modern LLM has thousands of weight matrices, each storable at one of several
hardware precision formats. AURA splits quantization into two questions and answers
the hard one by measurement:

- **Local (well studied):** given a fixed format, round this one matrix well —
  GPTQ, implicit clipping, activation-order. PrismaAURA runs the full deliberate
  render under every Linear.
- **Global (PrismaQuant's contribution):** how many bits should each Linear get,
  and in which format? AURA prices each `(Linear, format)` by a KL–Fisher quadratic
  — the second-order effect of that Linear's quantization error on the model's
  output distribution, measured with stochastic probes through the real model —
  and solves a multiple-choice knapsack over the bit budget. A heterogeneous
  per-Linear assignment extracts quality no single-format method structurally can.

## Artifact details

- Source model: `Qwen/Qwen3.6-27B`
- Export format: vLLM `compressed-tensors`, mixed precision
- Format menu: NVFP4 (group 16) + FP8 (E4M3 dynamic) + BF16, allocated per-Linear by AURA
- Target hardware: NVIDIA Blackwell (NVFP4-native)
- MTP tensors: included (BF16 passthrough)
- Size on disk: ~23 GB (~5.5 bpp over quantizable parameters)
- Passthrough dtype policy: source dtype preserved (no silent FP32 upcasting)

This is a quality-first operating point: it is **larger** than the 20.17 GB
PrismaSCOUT artifact, not smaller. A matched-footprint AURA point is in progress.
Downstream task evals (GSM8K / IFEval / MMLU) are forthcoming; the numbers above
are what has been directly measured.

## Serving

```bash
vllm serve rdtand/Qwen3.6-27B-PrismaAURA-5.5bit-vllm \
  --quantization compressed-tensors \
  --trust-remote-code \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'
```

---
PrismaQuant — mixed-precision LLM quantization that chooses the right format per
Linear on real end-to-end KL. Contact: robert.tand@icloud.com