---
library_name: transformers
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3.5-35B-A3B/blob/main/LICENSE
pipeline_tag: image-text-to-text
base_model:
- huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated
tags:
- abliterated
- uncensored
- NVFP4
---

# huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated-NVFP4

This is the NVFP4 quantitative version of [huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated](https://huggingface.co/huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated) created using [vllm-project/llm-compressor](https://github.com/vllm-project/llm-compressor/blob/main/examples/quantization_w4a4_fp4/qwen3_5_example.py)

### Note
This is just an attempt at NVFP4 quantization; no further tests have been conducted. If there are any issues, please leave a message.

# VLLM
### 1. Environmental installation

```
uv self update

cd /vllm/vllm-project/
mkdir venv
cd /vllm/vllm-project/venv

uv venv --python 3.12 --seed --managed-python

cd /vllm/vllm-project/venv
source .venv/bin/activate

cd /vllm/vllm-project
git clone ssh://git@github.com/vllm-project/vllm

cd /vllm/vllm-project/vllm
pip install -e .
```
### 2. Environment validation

```
uv pip show vllm
Version: 0.18.1rc1.dev115+gd6bb2a9d9.cu130

uv pip show compressed-tensors
Version: 0.14.0.1

uv pip show transformers
Version: 4.57.6

uv pip show torch
Version: 2.10.0+cu130

uv pip show llmcompressor
warning: Package(s) not found for: llmcompressor
```

### 3. Download the model.
```
cd /vllm/models

hf download huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated-NVFP4 --local-dir ./huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated-NVFP4 --token xxx

```

### 4. Load the model
```
cd /vllm/vllm-project/venv
source .venv/bin/activate

export CUDA_VISIBLE_DEVICES=0
unset LD_LIBRARY_PATH

export VLLM_USE_FLASHINFER_MOE_FP4=0
export VLLM_NVFP4_GEMM_BACKEND=cutlass

cd /vllm/models

vllm serve huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated-NVFP4 \
  --tensor-parallel-size 1 \
  --max-model-len 16384 \
  --trust-remote-code \
  --skip-mm-profiling \
  --limit-mm-per-prompt '{"image": 0, "video": 0}' \
  --enforce-eager \
  --gpu-memory-utilization 0.78 \
  --disable-custom-all-reduce
```

### 5. Simple test
```
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "huihui-ai/Huihui-Qwen3.5-35B-A3B-abliterated-NVFP4",
    "messages": [
      {"role": "user", "content": "hello"}
    ],
    "temperature": 0.7,
    "top_p": 0.9,
    "max_tokens": 8192,
    "stream": false
  }'
```


### 6. Issue
If you encounter the following issue:
ValueError: The Tokenizer class TokenizersBackend does not exist or is not currently imported.

**Modify tokenizer_config.json**

TokenizersBackend-> Qwen2Tokenizer

  "tokenizer_class": "TokenizersBackend", 
  
Change it to 

  "tokenizer_class": "Qwen2Tokenizer", 



### Usage Warnings


 - **Risk of Sensitive or Controversial Outputs**: This model’s safety filtering has been significantly reduced, potentially generating sensitive, controversial, or inappropriate content. Users should exercise caution and rigorously review generated outputs.

 - **Not Suitable for All Audiences**: Due to limited content filtering, the model’s outputs may be inappropriate for public settings, underage users, or applications requiring high security.

 - **Legal and Ethical Responsibilities**: Users must ensure their usage complies with local laws and ethical standards. Generated content may carry legal or ethical risks, and users are solely responsible for any consequences.

 - **Research and Experimental Use**: It is recommended to use this model for research, testing, or controlled environments, avoiding direct use in production or public-facing commercial applications.

 - **Monitoring and Review Recommendations**: Users are strongly advised to monitor model outputs in real-time and conduct manual reviews when necessary to prevent the dissemination of inappropriate content.

 - **No Default Safety Guarantees**: Unlike standard models, this model has not undergone rigorous safety optimization. huihui.ai bears no responsibility for any consequences arising from its use.


### Donation
##### Your donation helps us continue our further development and improvement, a cup of coffee can do it.
- bitcoin:
```
  bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge
```
- Support our work on [Ko-fi](https://ko-fi.com/huihuiai)!