trithemius/Qwen3.8-27B-heretic-ara-MTP-PrismAura-5.5bit

Qwen3.8-27B-heretic-ara — PrismAura 5.5bit (Blackwell, NVFP4 + FP8 + BF16, MTP)

PrismaQuant export of trohrbaugh/Qwen3.8-27B-heretic-ara for vLLM compressed-tensors serving on NVIDIA Blackwell.

This is a PrismAura artifact: a per-Linear mixed-precision allocation chosen by AURA — PrismaQuant's production-faithful KL–Fisher allocator — over the full NVFP4 / FP8 / BF16 format menu. Rather than quantizing every Linear to a single fixed format, AURA prices each (Linear, format) pair by its measured second-order effect on the model's output distribution and solves a bit-budget knapsack over the whole network, spending precision where the model is sensitive and saving it where it isn't. MTP (multi-token prediction) tensors are included and left in BF16, so the model can be served with vLLM's speculative decoding out of the box.

The recipe applies the proven PrismaAura 5.5bit quality-first allocation directly to trohrbaugh's decensored Arbitrary-Rank Ablation (ARA) build of Qwen3.8-27B.

What this model is

trohrbaugh/Qwen3.8-27B-heretic-ara is a decensored fine-tune/variant of Qwen3.8-27B processed using Heretic with the Arbitrary-Rank Ablation (ARA) method. ARA removes refusal and safety direction representations directly from the weight tensors rather than relying on system prompts or standard SFT adjustments.

This PrismAura build changes none of that behavior — it is a numerical re-encoding of the same weights into a mixed-precision compressed-tensors layout for efficient vLLM serving. It does not add, remove, or retrain anything at the weight level beyond quantization.

⚠️ This is a decensored, general-purpose model whose upstream tuning intentionally reduces refusal behavior, including for explicit, violent, and NSFW content. It carries no built-in content moderation. Deploy it behind your own usage policy and access controls if that matters for your use case.

Quantization details

• Source model: trohrbaugh/Qwen3.8-27B-heretic-ara
• Base architecture: Qwen/Qwen3.8-27B — 27B dense, causal LM with vision encoder, 64 hidden layers, 262,144-token native context
• Method: PrismaQuant / AURA (KL–Fisher-priced, per-Linear mixed-precision allocation; multiple-choice knapsack over a {NVFP4, FP8_E4M3, BF16} format menu)
• Local rounding: deliberate per-Linear render under every format (GPTQ-style with damping, activation-order handling, scale-faithful NVFP4 export)
• Export format: vLLM compressed-tensors, mixed precision
• Target hardware: NVIDIA Blackwell (NVFP4-native execution)
• MTP tensors: included, BF16 passthrough (8 Linears / 15 tensors)
• Vision tower: included, BF16 passthrough (110 Linears / 333 tensors uniformly assigned, not run through the KL–Fisher allocator)
• Passthrough dtype policy: source dtype preserved (no silent FP32 upcasting)
• Target bit budget: 5.5 bits per quantizable parameter

Allocator search (bit budget sweep)

AURA was run across a sweep of target bit budgets against the source model, tracking the achieved bits/parameter and the resulting KL–Fisher loss estimate at each point, along with how many "body" (non-visual, non-MTP) Linears landed in each format:

target bpp	achieved bpp	Δloss	NVFP4	FP8_E4M3	BF16
4.500	4.500	0.01777	303	1	0
4.600	4.600	0.01504	256	28	20
4.700	4.700	0.01400	245	36	23
4.750	4.750	0.01357	241	45	18
4.850	4.851	0.01282	224	52	28
5.000	5.001	0.01187	202	69	33
5.250	5.251	0.01053	183	94	27
5.500	5.501	0.00938	157	116	31
6.000	6.001	0.00738	120	152	32
7.000	7.000	0.00412	62	214	28
8.250	8.251	0.00162	1	246	57

This build uses the 5.500 → 5.501 operating point (bolded row above), identified by the solver as the raw-linear knee of the Rate-Distortion curve.

Final allocation

[alloc] target_bits=5.5: achieved_bits=5.501, Δloss=0.00938
[alloc] --visual-format=BF16: assigned 110 visual Linears uniformly
        (source=/gorgon/quant/qwen38-27b-heretic)
[alloc] --mtp-format=BF16: assigned 8 MTP Linears uniformly
[alloc] target=5.5 achieved=5.501

             NVFP4:   254 layers
          FP8_E4M3:   181 layers
              BF16:   179 layers

The body row above (157 / 116 / 31) reflects only the AURA-priced transformer Linears at the 5.5 target; the final totals (254 / 181 / 179) represent the entire network after the vision tower's 110 Linears and the 8 MTP Linears are added in as uniform BF16 passthrough alongside the rest of the quantizable Linears. The format menu includes an FP8 middle rung specifically so the allocator can spend extra bits on Linears that need it instead of jumping straight from NVFP4 to BF16.

Serving with vLLM

vllm serve trithemius/Qwen3.8-27B-heretic-ara-MTP-PrismAura-5.5bit \
  --quantization compressed-tensors \
  --trust-remote-code \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'

Notes:

• --quantization compressed-tensors is required to load the mixed NVFP4/FP8/BF16 format; NVFP4 execution requires an NVIDIA Blackwell GPU. On non-Blackwell hardware, vLLM falls back to dequantized execution.
• The --speculative-config MTP flags follow the base Qwen3.8 model card guidance for its qwen3_next_mtp speculator. Depending on your vLLM release, the method name may be "mtp".
• --max-model-len can be raised toward the model's native 262,144-token context if you have sufficient VRAM/KV-cache budget.
• Add --language-model-only if you wish to skip loading the vision tower on pure text workloads to free additional memory for KV cache.

What AURA does

A modern LLM has thousands of weight matrices, each of which could be stored at any of several hardware precision formats. AURA splits quantization into two questions and answers the hard one by measurement rather than heuristic:

• Local (well studied): given a fixed format, round this one matrix well — GPTQ, implicit clipping, activation-order. PrismaQuant runs a full deliberate render under every Linear for every candidate format.
• Global (PrismaQuant's contribution): how many bits should each Linear get, and in which format? AURA prices each (Linear, format) pair using a KL–Fisher quadratic — the second-order effect of that Linear's quantization error on the model's full output distribution, measured with stochastic probes through the real model — then solves a multiple-choice knapsack over the total bit budget. The result is a heterogeneous, per-Linear precision assignment that no single-format quantization method can produce structurally.

Disk size / model size

The total export size is ~22.1 GB across 5 safetensors shards. The underlying architecture is the full 27B-parameter Qwen3.8-27B model, compressed to ~5.5 bits per quantizable parameter on the body.

Acknowledgements

• Base architecture and instruct training: Qwen Team (Qwen/Qwen3.8-27B)
• Decensoring (Heretic, ARA method): trohrbaugh (trohrbaugh/Qwen3.8-27B-heretic-ara)
• PrismaQuant / AURA quantization method: rdtand (Qwen3.6-27B-PrismaAURA-5.5bit-vllm)
• This export: trithemius

License

Inherits apache-2.0 from the base model chain. You are responsible for complying with any additional usage terms from upstream contributors and for your own deployment's content policy, given the decensored nature of the source model.