README.md

---
base_model: Jackrong/Qwopus3.5-9B-v3
library_name: auto-round
tags:
  - quantized
  - auto-round
  - int3
  - w3a16
  - qwen3.5
  - coding
  - tool-calling
  - reasoning
license: apache-2.0
---

# Qwopus3.5-9B-v3 — W3A16 AutoRound (3-bit)

3-bit weight quantization of [Jackrong/Qwopus3.5-9B-v3](https://huggingface.co/Jackrong/Qwopus3.5-9B-v3) using [Intel AutoRound](https://github.com/intel/auto-round).

> **Note:** `lm_head` is **not quantized** (kept at bfloat16) due to a known vLLM incompatibility with quantized lm_head for the `qwen3_5` architecture. This adds ~1.9 GB but ensures correct loading in vLLM without patching.

## Model details

| Property | Value |
|----------|-------|
| Base model | Jackrong/Qwopus3.5-9B-v3 |
| Architecture | Qwen3.5-9B hybrid (DeltaNet + GatedAttention) |
| Quantization | W3A16 — 3-bit weights, 16-bit activations |
| Group size | 128 |
| Symmetric | Yes |
| lm_head | **Not quantized** (bfloat16) |
| Format | auto_round (auto_gptq packing) |
| Tool calling | ✅ hermes parser (`<tool_call>` format) |
| Reasoning | ✅ Qwen3 thinking mode |

## Quantization parameters

```python
AutoRound(
    scheme="W3A16",
    sym=True,
    group_size=128,
    iters=100,
    nsamples=22,
    seqlen=128,
    quant_lm_head=False,         # lm_head stays bfloat16
    quant_nontext_module=False,  # vision tower stays bfloat16
    layer_config={
        "mtp":    {"data_type": "bfloat16"},
        "mtp.fc": {"data_type": "bfloat16"},
    },
)
```

Calibration: Python, PHP, SQL, Bash, Docker, API patterns — domain-specific for coding agents.

## Memory requirements

| | |
|-|-|
| Model on disk | ~11 GB |
| VRAM (weights only) | ~6.5 GB |
| KV cache (12 GB GPU, util=0.93) | ~3.9 GB (~32k tokens) |

Fits on a single 12 GB VRAM GPU.

## Run with vLLM

```bash
vllm serve YOUR_USERNAME/Qwopus3.5-9B-W3A16-AutoRound \
    --served-model-name qwopus-9b \
    --port 8000 \
    --host 0.0.0.0 \
    --reasoning-parser qwen3 \
    --language-model-only \
    --max-model-len 65536 \
    --gpu-memory-utilization 0.93 \
    --max-num-seqs 16 \
    --max-num-batched-tokens 8192 \
    --enable-prefix-caching \
    --dtype half \
    --enable-auto-tool-choice \
    --tool-call-parser hermes
```

> Use `--tool-call-parser hermes` — the model outputs `<tool_call>` tags (Hermes format), not the `qwen3_coder` format.

## Usage

```python
from openai import OpenAI
import json

client = OpenAI(base_url="http://localhost:8000/v1", api_key="empty")

# Basic chat
response = client.chat.completions.create(
    model="qwopus-9b",
    messages=[{"role": "user", "content": "Write a Python async REST client."}],
    max_tokens=1024,
)
print(response.choices[0].message.content)

# Tool calling
tools = [{
    "type": "function",
    "function": {
        "name": "execute_code",
        "description": "Execute Python code and return output",
        "parameters": {
            "type": "object",
            "properties": {
                "code": {"type": "string"},
                "language": {"type": "string", "enum": ["python", "bash"]}
            },
            "required": ["code"]
        }
    }
}]

response = client.chat.completions.create(
    model="qwopus-9b",
    messages=[{"role": "user", "content": "Calculate fibonacci up to 10 terms."}],
    tools=tools,
    tool_choice="auto",
)

msg = response.choices[0].message
if msg.tool_calls:
    print(f"Tool: {msg.tool_calls[0].function.name}")
    print(f"Args: {msg.tool_calls[0].function.arguments}")
```

## Known limitations

- **lm_head not quantized** — vLLM's `Qwen3_5ForCausalLM` currently does not support quantized lm_head; kept at bfloat16
- Add `--enforce-eager` if you encounter CUDA graph issues