Instructions to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="bjk110/Qwen3.5-122B-A10B-abliterated-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForCausalLM

processor = AutoProcessor.from_pretrained("bjk110/Qwen3.5-122B-A10B-abliterated-FP8")
model = AutoModelForCausalLM.from_pretrained("bjk110/Qwen3.5-122B-A10B-abliterated-FP8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bjk110/Qwen3.5-122B-A10B-abliterated-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bjk110/Qwen3.5-122B-A10B-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bjk110/Qwen3.5-122B-A10B-abliterated-FP8

SGLang

How to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bjk110/Qwen3.5-122B-A10B-abliterated-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bjk110/Qwen3.5-122B-A10B-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bjk110/Qwen3.5-122B-A10B-abliterated-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bjk110/Qwen3.5-122B-A10B-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with Docker Model Runner:
```
docker model run hf.co/bjk110/Qwen3.5-122B-A10B-abliterated-FP8
```

Ban commited on Mar 28

Commit

e9434b3

verified ·

1 Parent(s): ee139d3

Update model card to English

Browse files

Files changed (1) hide show

README.md +54 -61

README.md CHANGED Viewed

@@ -14,40 +14,37 @@ license: apache-2.0
 # Qwen3.5-122B-A10B-abliterated-FP8
-[wangzhang/Qwen3.5-122B-A10B-abliterated](https://huggingface.co/wangzhang/Qwen3.5-122B-A10B-abliterated)의
-**FP8 E4M3 block-wise 양자화** 버전입니다.
-[Qwen/Qwen3.5-122B-A10B-FP8](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8)과 동일한 저장 포맷을 사용합니다.
-## 모델 정보
-| 항목 | 값 |
-|------|-----|
-| 원본 모델 | [wangzhang/Qwen3.5-122B-A10B-abliterated](https://huggingface.co/wangzhang/Qwen3.5-122B-A10B-abliterated) |
-| 양자화 | FP8 E4M3, block-wise 128×128, `weight_scale_inv` (bfloat16) |
-| 체크포인트 크기 | **115 GB** (원본 228 GB 대비 50% 감소) |
-| 활성 파라미터 | ~10B (MoE, 256 experts 중 10개 활성) |
-| 전체 파라미터 | 122B |
-| 최대 컨텍스트 | 32,768 tokens |
-| 검열 해제 | ✅ (abliterated, refusal rate 0.5%) |
-| Activation | Dynamic (체크포인트에 미포함) |
-| BF16 유지 대상 | lm_head, embed_tokens, norms, conv1d, router gate, in_proj_a/b |
-## 사용법
-### ⚠️ vLLM 패치 필요
-이 모델은 `qwen3_5_moe_text` (text-only MoE) 아키텍처를 사용합니다.
-vLLM v0.17~v0.18에서는 이 아키텍처가 정식 지원되지 않으므로,
-함께 제공되는 `vllm_patches/patch_qwen35_moe_text.py` 패치를 적용해야 합니다.
-### 서빙 (vLLM)
 ```bash
-# 1. 패치 적용 (vLLM 프로세스 시작 전에 실행)
 python vllm_patches/patch_qwen35_moe_text.py
-# 2. 서빙
 vllm serve /path/to/Qwen3.5-122B-A10B-abliterated-FP8 \
     --tensor-parallel-size 2 \
     --trust-remote-code \
@@ -56,57 +53,47 @@ vllm serve /path/to/Qwen3.5-122B-A10B-abliterated-FP8 \
     --reasoning-parser qwen3
 ```
-### Docker에서 패치 자동 적용
-entrypoint.sh 시작 부분에 추가:
 ```bash
 if [ -f /patches/patch_qwen35_moe_text.py ]; then
     python3 /patches/patch_qwen35_moe_text.py || true
 fi
 ```
-docker-compose.yml에 볼륨 마운트:
 ```yaml
 volumes:
   - ./vllm_patches:/patches:ro
 ```
-## 패치가 해결하는 문제
-| 문제 | 원인 | 해결 |
-|------|------|------|
-| `Qwen3_5MoeForCausalLM` 미인식 | vLLM registry 미등록 | TextOnlyShim 등록 |
-| Hybrid cache page-size 에러 | text-only CausalLM 경로 버그 | multimodal wrapper 경로 재사용 |
-| Vision encoder 초기화 실패 | wrapper가 vision 강제 초기화 | vision encoder 스킵 |
-| TP2 block_k 에러 | vision hidden_size=1152 참조 | dummy vision config 주입 |
-vLLM이 `qwen3_5_moe_text`를 정식 지원하면 패치가 불필요해집니다.
-## 벤치마크 (한국어 QA 12문항, DGX Spark TP=2)
-| 지표 | 공식 Qwen FP8 | **본 모델** | BF16+Runtime FP8 |
-|------|:---:|:---:|:---:|
-| 완답 | 12/12 | **12/12** | 11/12 |
-| 속도 | 20.3 tok/s | **29.6 tok/s** | 29.3 tok/s |
-| 크기 | 119 GB | **115 GB** | 228 GB |
-| Swap 필요 | No | **No** | Yes (200GB) |
-| 검열 해제 | ❌ | **✅** | ✅ |
-Runtime FP8(BF16 원본 + `--quantization fp8`)과 동일한 출력 품질이면서,
-체크포인트 크기 50% 감소 + swap 불필요 + 빠른 로딩.
-## 하드웨어 요구사항
-| 구성 | GPU 메모리 | 비고 |
-|------|-----------|------|
-| TP=1 | ~115 GB | H100 80GB 불가, GB200 등 |
-| **TP=2** | **~58 GB/GPU** | DGX Spark, H100×2, A100 80GB×2 |
-| TP=4 | ~29 GB/GPU | A100 40GB×4 |
-## 양자화 방법
-calibration 없이 BF16 weight를 128×128 block 단위로 직접 FP8 E4M3 변환합니다.
-[fp8-quantizer](https://github.com/JungkwanBan/fp8-quantizer)로 생성.
 ```bash
 python convert_bf16_to_fp8.py \
@@ -114,12 +101,18 @@ python convert_bf16_to_fp8.py \
     --output ./Qwen3.5-122B-A10B-abliterated-FP8
 ```
-## 원본 모델 정보
-[wangzhang/Qwen3.5-122B-A10B-abliterated](https://huggingface.co/wangzhang/Qwen3.5-122B-A10B-abliterated):
-[Prometheus](https://github.com/wuwangzhang1216/prometheus)를 사용한 검열 해제 버전.
-Refusal rate 0.5% (200개 테스트 중 1개), KL divergence 0.0115.
-## 라이선스
-원본 모델의 라이선스를 따릅니다.

 # Qwen3.5-122B-A10B-abliterated-FP8
+**FP8 E4M3 block-wise quantized** version of [wangzhang/Qwen3.5-122B-A10B-abliterated](https://huggingface.co/wangzhang/Qwen3.5-122B-A10B-abliterated).
+Same checkpoint format as [Qwen/Qwen3.5-122B-A10B-FP8](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8).
+## Model Details
+| | |
+|---|---|
+| Base Model | [wangzhang/Qwen3.5-122B-A10B-abliterated](https://huggingface.co/wangzhang/Qwen3.5-122B-A10B-abliterated) |
+| Quantization | FP8 E4M3, block-wise 128×128, `weight_scale_inv` (bfloat16) |
+| Checkpoint Size | **115 GB** (50% of original 228 GB) |
+| Active Parameters | ~10B (MoE, 10 of 256 experts active per token) |
+| Total Parameters | 122B |
+| Max Context | 32,768 tokens |
+| Uncensored | ✅ (abliterated, 0.5% refusal rate) |
+| Activation | Dynamic (not stored in checkpoint) |
+| BF16 Preserved | lm_head, embed_tokens, norms, conv1d, router gate, in_proj_a/b |
+## Usage
+### ⚠️ vLLM Patch Required
+This model uses the `qwen3_5_moe_text` (text-only MoE) architecture, which is not natively supported in vLLM v0.17–v0.18. You must apply the included `vllm_patches/patch_qwen35_moe_text.py` before serving.
+### Serving with vLLM
 ```bash
+# 1. Apply patch (run before vLLM process starts)
 python vllm_patches/patch_qwen35_moe_text.py
+# 2. Serve
 vllm serve /path/to/Qwen3.5-122B-A10B-abliterated-FP8 \
     --tensor-parallel-size 2 \
     --trust-remote-code \
     --reasoning-parser qwen3
 ```
+### Docker Entrypoint Auto-Patch
+Add to the beginning of your entrypoint.sh:
 ```bash
 if [ -f /patches/patch_qwen35_moe_text.py ]; then
     python3 /patches/patch_qwen35_moe_text.py || true
 fi
 ```
+Mount the patches volume in docker-compose.yml:
 ```yaml
 volumes:
   - ./vllm_patches:/patches:ro
 ```
+## What the Patch Does
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| `Qwen3_5MoeForCausalLM` not recognized | Not in vLLM registry | Registers TextOnlyShim class |
+| Hybrid cache page-size error | Bug in text-only CausalLM path | Reuses multimodal wrapper's cache-spec |
+| Vision encoder init failure | Wrapper forces vision init | Skips vision encoder |
+| TP2 block_k=128 error | vision_config.hidden_size=1152 | Injects dummy vision config |
+The patch will become unnecessary once vLLM adds native support for `qwen3_5_moe_text`.
+## Benchmark (Korean QA, 12 questions, DGX Spark TP=2)
+| Metric | Official Qwen FP8 | **This Model** | BF16 + Runtime FP8 |
+|--------|:--:|:--:|:--:|
+| Completed | 12/12 | **12/12** | 11/12 |
+| Speed | 20.3 tok/s | **29.6 tok/s** | 29.3 tok/s |
+| Size | 119 GB | **115 GB** | 228 GB |
+| Swap Required | No | **No** | Yes (200GB) |
+| Uncensored | ❌ | **✅** | ✅ |
+Same output quality as runtime FP8, with half the checkpoint size, no swap needed, and faster loading.
+## Quantization Method
+Block-wise FP8 E4M3 conversion without calibration — computes per-block max as scale. Built with [fp8-quantizer](https://github.com/JungkwanBan/fp8-quantizer).
 ```bash
 python convert_bf16_to_fp8.py \
     --output ./Qwen3.5-122B-A10B-abliterated-FP8
 ```
+## Hardware Requirements
+| Config | GPU Memory | Notes |
+|--------|-----------|-------|
+| TP=1 | ~115 GB | Requires GB200 or similar |
+| **TP=2** | **~58 GB/GPU** | DGX Spark, H100×2, A100 80GB×2 |
+| TP=4 | ~29 GB/GPU | A100 40GB×4 |
+## Base Model
+[wangzhang/Qwen3.5-122B-A10B-abliterated](https://huggingface.co/wangzhang/Qwen3.5-122B-A10B-abliterated) — uncensored via [Prometheus](https://github.com/wuwangzhang1216/prometheus) abliteration. Refusal rate 0.5% (1/200), KL divergence 0.0115.
+## License
+Follows the license of the base model.