Instructions to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="bjk110/Qwen3.5-122B-A10B-abliterated-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForCausalLM

processor = AutoProcessor.from_pretrained("bjk110/Qwen3.5-122B-A10B-abliterated-FP8")
model = AutoModelForCausalLM.from_pretrained("bjk110/Qwen3.5-122B-A10B-abliterated-FP8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bjk110/Qwen3.5-122B-A10B-abliterated-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bjk110/Qwen3.5-122B-A10B-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bjk110/Qwen3.5-122B-A10B-abliterated-FP8

SGLang

How to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bjk110/Qwen3.5-122B-A10B-abliterated-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bjk110/Qwen3.5-122B-A10B-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bjk110/Qwen3.5-122B-A10B-abliterated-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bjk110/Qwen3.5-122B-A10B-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with Docker Model Runner:
```
docker model run hf.co/bjk110/Qwen3.5-122B-A10B-abliterated-FP8
```

Qwen3.5-122B-A10B-abliterated-FP8 / README.md

Ban

Upload FP8 block-wise quantized model (128x128, weight_scale_inv)

ee139d3 verified 5 months ago

4.14 kB

base_model: wangzhang/Qwen3.5-122B-A10B-abliterated
pipeline_tag: text-generation
tags:
  - qwen3.5
  - moe
  - fp8
  - quantized
  - abliterated
  - uncensored
  - block-wise-fp8
license: apache-2.0

Qwen3.5-122B-A10B-abliterated-FP8

wangzhang/Qwen3.5-122B-A10B-abliterated의 FP8 E4M3 block-wise 양자화 버전입니다.

Qwen/Qwen3.5-122B-A10B-FP8과 동일한 저장 포맷을 사용합니다.

모델 정보

항목	값
원본 모델	wangzhang/Qwen3.5-122B-A10B-abliterated
양자화	FP8 E4M3, block-wise 128×128, `weight_scale_inv` (bfloat16)
체크포인트 크기	115 GB (원본 228 GB 대비 50% 감소)
활성 파라미터	~10B (MoE, 256 experts 중 10개 활성)
전체 파라미터	122B
최대 컨텍스트	32,768 tokens
검열 해제	✅ (abliterated, refusal rate 0.5%)
Activation	Dynamic (체크포인트에 미포함)
BF16 유지 대상	lm_head, embed_tokens, norms, conv1d, router gate, in_proj_a/b

사용법

⚠️ vLLM 패치 필요

이 모델은 qwen3_5_moe_text (text-only MoE) 아키텍처를 사용합니다. vLLM v0.17~v0.18에서는 이 아키텍처가 정식 지원되지 않으므로, 함께 제공되는 vllm_patches/patch_qwen35_moe_text.py 패치를 적용해야 합니다.

서빙 (vLLM)

# 1. 패치 적용 (vLLM 프로세스 시작 전에 실행)
python vllm_patches/patch_qwen35_moe_text.py

# 2. 서빙
vllm serve /path/to/Qwen3.5-122B-A10B-abliterated-FP8 \
    --tensor-parallel-size 2 \
    --trust-remote-code \
    --enable-chunked-prefill \
    --max-model-len 32768 \
    --reasoning-parser qwen3

Docker에서 패치 자동 적용

entrypoint.sh 시작 부분에 추가:

if [ -f /patches/patch_qwen35_moe_text.py ]; then
    python3 /patches/patch_qwen35_moe_text.py || true
fi

docker-compose.yml에 볼륨 마운트:

volumes:
  - ./vllm_patches:/patches:ro

패치가 해결하는 문제

문제	원인	해결
`Qwen3_5MoeForCausalLM` 미인식	vLLM registry 미등록	TextOnlyShim 등록
Hybrid cache page-size 에러	text-only CausalLM 경로 버그	multimodal wrapper 경로 재사용
Vision encoder 초기화 실패	wrapper가 vision 강제 초기화	vision encoder 스킵
TP2 block_k 에러	vision hidden_size=1152 참조	dummy vision config 주입

vLLM이 qwen3_5_moe_text를 정식 지원하면 패치가 불필요해집니다.

벤치마크 (한국어 QA 12문항, DGX Spark TP=2)

지표	공식 Qwen FP8	본 모델	BF16+Runtime FP8
완답	12/12	12/12	11/12
속도	20.3 tok/s	29.6 tok/s	29.3 tok/s
크기	119 GB	115 GB	228 GB
Swap 필요	No	No	Yes (200GB)
검열 해제	❌	✅	✅

Runtime FP8(BF16 원본 + --quantization fp8)과 동일한 출력 품질이면서, 체크포인트 크기 50% 감소 + swap 불필요 + 빠른 로딩.

하드웨어 요구사항

구성	GPU 메모리	비고
TP=1	~115 GB	H100 80GB 불가, GB200 등
TP=2	~58 GB/GPU	DGX Spark, H100×2, A100 80GB×2
TP=4	~29 GB/GPU	A100 40GB×4

양자화 방법

calibration 없이 BF16 weight를 128×128 block 단위로 직접 FP8 E4M3 변환합니다. fp8-quantizer로 생성.

python convert_bf16_to_fp8.py \
    --model wangzhang/Qwen3.5-122B-A10B-abliterated \
    --output ./Qwen3.5-122B-A10B-abliterated-FP8

원본 모델 정보

wangzhang/Qwen3.5-122B-A10B-abliterated: Prometheus를 사용한 검열 해제 버전. Refusal rate 0.5% (200개 테스트 중 1개), KL divergence 0.0115.

라이선스

원본 모델의 라이선스를 따릅니다.