README.md

---
license: apache-2.0
base_model:
  - huihui-ai/Huihui-Qwen3-30B-A3B-Thinking-2507-abliterated
base_model_relation: quantized
library_name: transformers
pipeline_tag: text-generation
tags:
  - safetensors
  - qwen3
  - moe
  - nvfp4
  - 4-bit
  - quantized
  - abliterated
  - dgx-spark
  - blackwell
  - gb10
  - sm121
  - vllm
  - llm-compressor
language:
  - en
  - zh
---

# Huihui-Qwen3-30B-A3B-Thinking-2507-abliterated-NVFP4

[English](#english) | [繁體中文](#繁體中文)

---

## English

> [!TIP]
> **Re-quantized on 2026-04-13** with corrected ignore list (`mlp.gate` + `embed_tokens` now preserved in BF16), fixing routing quality issues in the previous release.

> [!WARNING]
> **NVIDIA DGX Spark (GB10 SM121) — Driver 590.48+ / CUDA 13.1+**
>
> As of April 2026, NVFP4 software support on SM121 is still incomplete. The native W4A4 compute path is not yet functional on this hardware — the runtime silently falls back to W4A16 (BF16 activations), negating the theoretical throughput advantage of FP4.
>
> If **accuracy and inference speed** are your priority, we recommend the INT4 AutoRound version:
> 👉 **[YuYu1015/Huihui-Qwen3.5-35B-A3B-Claude-4.6-Opus-abliterated-int4-AutoRound](https://huggingface.co/YuYu1015/Huihui-Qwen3.5-35B-A3B-Claude-4.6-Opus-abliterated-int4-AutoRound)**
>
> INT4 AutoRound leverages the mature W4A16 Marlin kernel path on DGX Spark, offering more thorough calibration (~99.5% quality retention) and significantly more stable performance. The full potential of NVFP4 will only be unlocked once NVIDIA delivers complete W4A4 kernel support for SM121.

NVFP4 quantization of [huihui-ai/Huihui-Qwen3-30B-A3B-Thinking-2507-abliterated](https://huggingface.co/huihui-ai/Huihui-Qwen3-30B-A3B-Thinking-2507-abliterated), optimized for **NVIDIA DGX Spark (GB10 SM121)**.

### Model Details

| Item           | Value                                                                        |
| -------------- | ---------------------------------------------------------------------------- |
| Architecture   | MoE (30B total, 3B active), 48 layers, 128 experts, top-8 routing            |
| Base model     | [Qwen/Qwen3-30B-A3B](https://huggingface.co/Qwen/Qwen3-30B-A3B)              |
| Fine-tuned by  | [huihui-ai](https://huggingface.co/huihui-ai) (Thinking 2507 + abliteration) |
| Quantized by   | [YuYu1015](https://huggingface.co/YuYu1015)                                  |
| Model size     | ~18.1 GB (NVFP4, vs ~60 GB BF16 original)                                    |
| Context length | Up to 131,072 tokens                                                         |
| Thinking mode  | Built-in Chain-of-Thought reasoning (enabled by default)                     |
| Tool calling   | Supported (`qwen3_coder` parser)                                             |

### Quantization Details

| Item                        | Value                                                                                                            |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Method                      | [llm-compressor](https://github.com/vllm-project/llm-compressor) v0.10.0.1                                       |
| Scheme                      | NVFP4 (E2M1 + FP8 per-group scaling, group size 16)                                                              |
| Format                      | compressed-tensors v0.14.0.1                                                                                     |
| Calibration dataset         | [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) (`train_sft` split) |
| Calibration samples         | 512                                                                                                              |
| Calibration sequence length | 2048                                                                                                             |
| MoE expert calibration      | `moe_calibrate_all_experts=True` (all experts receive calibration data)                                          |
| Hardware                    | NVIDIA DGX Spark (GB10, 128GB unified memory)                                                                    |
| Environment                 | `transformers==4.57.1` + `llm-compressor==0.10.0.1`                                                              |

### Layers Preserved in BF16

The following layers are **not quantized** to preserve model quality:

| Layer                | Reason                                                        |
| -------------------- | ------------------------------------------------------------- |
| `lm_head`            | Output head, sensitive to quantization noise                  |
| `re:.*mlp.gate$`     | **MoE routing gate** — critical for expert selection accuracy |
| `re:.*embed_tokens$` | Input embeddings                                              |

### Serving with vLLM

```bash
vllm serve /path/to/model \
    --quantization compressed-tensors \
    --served-model-name qwen3-30b \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 32768 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --trust-remote-code
```

### DGX Spark (SM121) Compatibility Notes

- NVFP4 on SM121 falls back to W4A16 (native W4A4 path not yet supported, missing `cvt.e2m1x2` instruction)
- Qwen3 (non-3.5) has no Mamba layers, so FP8 KV cache works safely
- Qwen3 has no GDN, so `linear_attn` does not need to be excluded
- Clear page cache before starting on UMA: `sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'`

### Safety Warning

This model has safety filtering removed (abliterated) and may generate inappropriate content. Users are solely responsible for all consequences arising from its use.

### Credits

- **Original Model**: [Qwen/Qwen3-30B-A3B](https://huggingface.co/Qwen/Qwen3-30B-A3B) by Alibaba Qwen Team
- **Thinking 2507 & Abliteration**: [huihui-ai](https://huggingface.co/huihui-ai)
- **NVFP4 Quantization**: [YuYu1015](https://huggingface.co/YuYu1015) on NVIDIA DGX Spark (GB10)
- **Quantization Tool**: [llm-compressor](https://github.com/vllm-project/llm-compressor) by vLLM Project
- **Reference**: [RedHatAI/Qwen3-30B-A3B-NVFP4](https://huggingface.co/RedHatAI/Qwen3-30B-A3B-NVFP4)

---

## 繁體中文

> [!TIP]
> **2026-04-13 重新量化上傳**，修正先前版本的 ignore list（`mlp.gate` 與 `embed_tokens` 現在保留 BF16），解決 MoE 路由品質問題。

> [!WARNING]
> **NVIDIA DGX Spark (GB10 SM121) 使用者 — Driver 590.48+ / CUDA 13.1+**
>
> 截至 2026 年 4 月，NVFP4 在 SM121 上的軟體支援仍不完整。原生 W4A4 運算路徑尚未在此硬體上就緒——執行時會靜默退回 W4A16（BF16 activation），FP4 的理論吞吐量優勢無法發揮。
>
> 若**精度與推理速度**為首要考量，建議改用 INT4 AutoRound 版本：
> 👉 **[YuYu1015/Huihui-Qwen3.5-35B-A3B-Claude-4.6-Opus-abliterated-int4-AutoRound](https://huggingface.co/YuYu1015/Huihui-Qwen3.5-35B-A3B-Claude-4.6-Opus-abliterated-int4-AutoRound)**
>
> INT4 AutoRound 在 DGX Spark 上使用成熟的 W4A16 Marlin kernel 路徑，校準更完整（品質保留約 99.5%），效能顯著更穩定。待 NVIDIA 為 SM121 提供完整的 W4A4 kernel 支援後，NVFP4 的真正優勢才能發揮。

[huihui-ai/Huihui-Qwen3-30B-A3B-Thinking-2507-abliterated](https://huggingface.co/huihui-ai/Huihui-Qwen3-30B-A3B-Thinking-2507-abliterated) 的 NVFP4 量化版本，針對 **NVIDIA DGX Spark (GB10 SM121)** 最佳化。

### 模型資訊

| 項目         | 數值                                                                          |
| ------------ | ----------------------------------------------------------------------------- |
| 架構         | MoE（30B 總參數, 3B 活躍），48 層，128 experts，top-8 routing                 |
| 基礎模型     | [Qwen/Qwen3-30B-A3B](https://huggingface.co/Qwen/Qwen3-30B-A3B)               |
| 微調者       | [huihui-ai](https://huggingface.co/huihui-ai)（Thinking 2507 + abliteration） |
| 量化者       | [YuYu1015](https://huggingface.co/YuYu1015)                                   |
| 模型大小     | ~18.1 GB（NVFP4，原版 BF16 約 60 GB）                                         |
| Context 長度 | 最高 131,072 tokens                                                           |
| 思考模式     | 內建思維鏈推理（預設啟用）                                                    |
| 工具呼叫     | 支援（`qwen3_coder` parser）                                                  |

### 量化詳情

| 項目         | 數值                                                                                                            |
| ------------ | --------------------------------------------------------------------------------------------------------------- |
| 方法         | [llm-compressor](https://github.com/vllm-project/llm-compressor) v0.10.0.1                                      |
| 方案         | NVFP4（E2M1 + FP8 逐群縮放，群組大小 16）                                                                       |
| 格式         | compressed-tensors v0.14.0.1                                                                                    |
| 校準資料集   | [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) (`train_sft` 分割) |
| 校準樣本數   | 512                                                                                                             |
| 校準序列長度 | 2048                                                                                                            |
| MoE 專家校準 | `moe_calibrate_all_experts=True`（所有專家都接收校準資料）                                                      |
| 量化硬體     | NVIDIA DGX Spark（GB10, 128GB 統一記憶體）                                                                      |
| 環境         | `transformers==4.57.1` + `llm-compressor==0.10.0.1`                                                             |

### 保留 BF16 的層

以下層**未被量化**以保持模型品質：

| 層                   | 原因                                   |
| -------------------- | -------------------------------------- |
| `lm_head`            | 輸出頭，對量化雜訊敏感                 |
| `re:.*mlp.gate$`     | **MoE 路由閘**——對專家選擇精度至關重要 |
| `re:.*embed_tokens$` | 輸入嵌入                               |

### vLLM 部署

```bash
vllm serve /path/to/model \
    --quantization compressed-tensors \
    --served-model-name qwen3-30b \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 32768 \
    --enable-prefix-caching \
    --enable-chunked-prefill \
    --trust-remote-code
```

### DGX Spark (SM121) 相容性說明

- NVFP4 在 SM121 上會退回 W4A16（原生 W4A4 路徑尚未支援，缺少 `cvt.e2m1x2` 指令）
- Qwen3（非 3.5）沒有 Mamba 層，FP8 KV cache 可以安全使用
- Qwen3 沒有 GDN，`linear_attn` 不需要排除
- UMA 架構啟動前請先清除 page cache：`sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'`

### 安全警告

此模型已移除安全過濾機制（abliterated），可能產生不當內容。使用者須自行承擔所有風險與法律責任。

### 致謝

- **原始模型**：[Qwen/Qwen3-30B-A3B](https://huggingface.co/Qwen/Qwen3-30B-A3B)，Alibaba Qwen 團隊
- **Thinking 2507 與去審查**：[huihui-ai](https://huggingface.co/huihui-ai)
- **NVFP4 量化**：[YuYu1015](https://huggingface.co/YuYu1015)，於 NVIDIA DGX Spark (GB10) 上完成
- **量化工具**：[llm-compressor](https://github.com/vllm-project/llm-compressor)，vLLM Project
- **參考**：[RedHatAI/Qwen3-30B-A3B-NVFP4](https://huggingface.co/RedHatAI/Qwen3-30B-A3B-NVFP4)