Huihui-Qwopus3.5-27B-v3-abliterated-NVFP4

NVFP4 quantized version of huihui-ai/Huihui-Qwopus3.5-27B-v3-abliterated — an abliterated Qwen 3.5 27B distilled from Claude Opus reasoning patterns (Jackrong/Qwopus3.5-27B-v3).

~51 GB → 19.8 GB. Single NVIDIA Blackwell GPU.

What Makes This Different

Qwopus = Qwen 3.5 + Opus-style reasoning. The base model was trained to produce stable, structured reasoning even at short output lengths — where vanilla Qwen3.5 needs more tokens to reach peak quality.

Characteristic	Vanilla Qwen3.5	Qwopus
Short responses (64-256 tok)	Quality ramps up slowly	Stable from the start
Long responses (2048+ tok)	Slightly higher peak	Marginally lower peak
Reasoning style	Variable	Structured "thinking process" chains
Best for	Long-form analysis	Agents, tool calls, code generation

This makes Qwopus especially suited for agentic workflows where each tool call or code snippet needs to be precise in a short response.

Key Specs

Base model	huihui-ai/Huihui-Qwopus3.5-27B-v3-abliterated
Architecture	Qwen 3.5 Dense — 27B parameters, 64 layers
Quantization	NVFP4 W4A4 (weights FP4, activations FP4, scales FP8)
Format	compressed-tensors (native vLLM support)
Tool	vllm-project/llm-compressor (main)
Calibration	512 samples, neuralmagic/calibration, seq_len=4096
Size	19.8 GB
Max context	262,144 tokens
MTP	Not available (removed during fine-tuning)
Requires	NVIDIA Blackwell GPU (SM 120), vLLM nightly (cu130)

Quickstart

vLLM

vllm serve Lna-Lab/Huihui-Qwopus3.5-27B-v3-abliterated-NVFP4 \
    --max-model-len 32768 \
    --reasoning-parser qwen3

With tool calling

vllm serve Lna-Lab/Huihui-Qwopus3.5-27B-v3-abliterated-NVFP4 \
    --max-model-len 32768 \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml

Docker

docker run --gpus '"device=0"' -p 8016:8016 \
    -v /path/to/model:/models/current:ro \
    --shm-size 16gb \
    vllm/vllm-openai:cu130-nightly \
    vllm serve /models/current --port 8016 --max-model-len 32768 \
    --reasoning-parser qwen3

Benchmark

Single NVIDIA RTX PRO 6000 Blackwell (96 GB VRAM).

Test	Tokens	Speed	Result
Short (derivative)	64	57.7 tok/s	PASS — structured reasoning even at 64 tokens
Short (CAP theorem)	128	58.5 tok/s	PASS
Mid (LRU cache)	512	57.6 tok/s	PASS
Long (architecture analysis)	1710	59.2 tok/s	PASS
Container burst	256	61.1 tok/s	PASS — 3 runs stable

Sustained: ~59-61 tok/s (single GPU).

Quantization Details

Recipe

recipe = QuantizationModifier(
    targets=["Linear"],
    ignore=["lm_head", "re:.*visual.*", "re:.*in_proj_a$", "re:.*in_proj_b$"],
    scheme="NVFP4",
)

Calibration

• Dataset: neuralmagic/calibration (LLM split)
• Samples: 512
• Max sequence length: 4096

Reproduction

from transformers import Qwen3_5ForConditionalGeneration, AutoProcessor, AutoTokenizer
from datasets import load_dataset
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
import torch

MODEL_ID = "huihui-ai/Huihui-Qwopus3.5-27B-v3-abliterated"
OUTPUT = "Huihui-Qwopus3.5-27B-v3-abliterated-NVFP4"

model = Qwen3_5ForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto", trust_remote_code=True)
processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)

recipe = QuantizationModifier(
    targets=["Linear"],
    ignore=["lm_head", "re:.*visual.*", "re:.*in_proj_a$", "re:.*in_proj_b$"],
    scheme="NVFP4",
)

ds = load_dataset("neuralmagic/calibration", name="LLM", split="train[:512]")

def preprocess(example):
    messages = [
        {"role": m["role"], "content": [{"type": "text", "text": m["content"]}]}
        for m in example["messages"]
    ]
    return processor.apply_chat_template(
        messages, return_tensors="pt", padding=False, truncation=True,
        max_length=4096, tokenize=True, add_special_tokens=False,
        return_dict=True, add_generation_prompt=False,
    )

ds = ds.map(preprocess, batched=False, remove_columns=ds.column_names)

def data_collator(batch):
    assert len(batch) == 1
    return {
        key: (torch.tensor(value) if key != "pixel_values"
              else torch.tensor(value, dtype=torch.bfloat16).squeeze(0))
        for key, value in batch[0].items()
    }

oneshot(
    model=model, recipe=recipe, dataset=ds,
    max_seq_length=4096, num_calibration_samples=512,
    data_collator=data_collator,
)

model.save_pretrained(OUTPUT, save_compressed=True)
processor.save_pretrained(OUTPUT)

Note: No save_mtp_tensors_to_checkpoint needed — Qwopus does not have MTP head (removed during fine-tuning).

Environment

Package	Version
torch	2.11.0+cu130
transformers	5.5.4
llmcompressor	0.1.dev (main @ 3084520)
compressed-tensors	0.15.1a20260414
CUDA	13.0

Requirements

• GPU: NVIDIA Blackwell (SM 120)
• VRAM: ~20 GB minimum
• Software: vLLM nightly (cu130)

Notes

• Abliterated (uncensored). Use responsibly.
• Vision tower preserved in BF16.
• No MTP head — speculative decoding not available for this variant.
• NVFP4 is Blackwell-specific. Will not work on Ampere/Hopper.

Credits

• Abliteration: huihui-ai
• Opus distillation: Jackrong/Qwopus3.5-27B-v3
• Original: Qwen
• Quantization recipe: lyf/HauhauCS (proven path)
• Quantization tool: vllm-project/llm-compressor

Support the Base Model Author

• Ko-fi: https://ko-fi.com/huihuiai
• Bitcoin: bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge