Instructions to use trohrbaugh/Qwen3.5-35B-A3B-heretic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use trohrbaugh/Qwen3.5-35B-A3B-heretic with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="trohrbaugh/Qwen3.5-35B-A3B-heretic")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("trohrbaugh/Qwen3.5-35B-A3B-heretic")
model = AutoModelForMultimodalLM.from_pretrained("trohrbaugh/Qwen3.5-35B-A3B-heretic", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use trohrbaugh/Qwen3.5-35B-A3B-heretic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "trohrbaugh/Qwen3.5-35B-A3B-heretic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trohrbaugh/Qwen3.5-35B-A3B-heretic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/trohrbaugh/Qwen3.5-35B-A3B-heretic

SGLang

How to use trohrbaugh/Qwen3.5-35B-A3B-heretic with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "trohrbaugh/Qwen3.5-35B-A3B-heretic" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trohrbaugh/Qwen3.5-35B-A3B-heretic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "trohrbaugh/Qwen3.5-35B-A3B-heretic" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "trohrbaugh/Qwen3.5-35B-A3B-heretic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use trohrbaugh/Qwen3.5-35B-A3B-heretic with Docker Model Runner:
```
docker model run hf.co/trohrbaugh/Qwen3.5-35B-A3B-heretic
```

trohrbaugh commited on May 2

Commit

d6adf83

verified ·

1 Parent(s): a0da17d

aux-rescue+moe-fuse: refresh banner

Browse files

Files changed (1) hide show

README.md +31 -11

README.md CHANGED Viewed

@@ -21,7 +21,7 @@ tags:
 > registered in the loaded model class's `state_dict()` were silently
 > dropped on save.
 >
-> **Symptom:** Affected workflows that depend on these tensors will fail to load. Other workflows were unaffected.
 >
 > **Status:** fixed on 2026-05-02 by uploading `model-auxiliary.safetensors` and
 > patching `model.safetensors.index.json`. The repo now carries an
@@ -35,6 +35,36 @@ tags:
 <!-- aux-rescue-banner: end -->
 # This is a decensored version of [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B), made using [Heretic](https://github.com/p-e-w/heretic) v1.2.0
 ## Abliteration parameters
@@ -60,7 +90,6 @@ tags:
 -----
 # Qwen3.5-35B-A3B
 <img width="400px" src="https://qianwen-res.oss-accelerate.aliyuncs.com/logo_qwen3.5.png">
@@ -94,12 +123,10 @@ Qwen3.5 features the following enhancement:
 - **Next-Generation Training Infrastructure**: Near-100% multimodal training efficiency compared to text-only training and asynchronous RL frameworks supporting massive-scale agent scaffolds and environment orchestration.
 ![Benchmark Results](https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3.5/Figures/qwen3.5_middle_size_score.png)
 For more details, please refer to our blog post [Qwen3.5](https://qwen.ai/blog?id=qwen3.5).
 ## Model Overview
 - Type: Causal Language Model with Vision Encoder
@@ -125,7 +152,6 @@ For more details, please refer to our blog post [Qwen3.5](https://qwen.ai/blog?i
     - MTP: trained with multi-steps
 - Context Length: 262,144 natively and extensible up to 1,010,000 tokens.
 ## Benchmark Results
 ### Language
@@ -951,7 +977,6 @@ For more details, please refer to our blog post [Qwen3.5](https://qwen.ai/blog?i
 > Qwen3.5 models operate in thinking mode by default, generating thinking content signified by `<think>\n...</think>\n\n` before producing the final responses.
 > To disable thinking content and obtain direct response, refer to the examples [here](#instruct-or-non-thinking-mode).
 For streamlined integration, we recommend using Qwen3.5 via APIs. Below is a guide to use Qwen3.5 via OpenAI-compatible API.
 ### Serving Qwen3.5
@@ -959,7 +984,6 @@ For streamlined integration, we recommend using Qwen3.5 via APIs. Below is a gui
 Qwen3.5 can be served via APIs with popular inference frameworks.
 In the following, we show example commands to launch OpenAI-Compatible API servers for Qwen3.5 models.
 > [!Important]
 > Inference efficiency and throughput vary significantly across frameworks.
 > We recommend using the latest framework versions to ensure optimal performance and compatibility.
@@ -1103,7 +1127,6 @@ chat_response = client.chat.completions.create(
 print("Chat response:", chat_response)
 ```
 #### Image Input
 ```python
@@ -1239,7 +1262,6 @@ print("Chat response:", chat_response)
 > [!Note]
 > If you are using APIs from Alibaba Cloud Model Studio, in addition to changing `model`, please use `"enable_thinking": False` instead of `"chat_template_kwargs": {"enable_thinking": False}`.
 ## Agentic Usage
 Qwen3.5 excels in tool calling capabilities.
@@ -1319,7 +1341,6 @@ print(responses)
 ### Qwen Code
 [Qwen Code](https://github.com/QwenLM/qwen-code) is an open-source AI agent for the terminal, optimized for Qwen models. It helps you understand large codebases, automate tedious work, and ship faster.
 For more information, please refer to [Qwen Code](https://qwenlm.github.io/qwen-code-docs/).
@@ -1398,7 +1419,6 @@ To achieve optimal performance, we recommend the following settings:
     Alternatively, override the default values via engine startup parameters. For implementation details, refer to: [vLLM](https://github.com/vllm-project/vllm/pull/34330) / [SGLang](https://github.com/sgl-project/sglang/pull/18467).
 ### Citation
 If you find our work helpful, feel free to give us a cite.

 > registered in the loaded model class's `state_dict()` were silently
 > dropped on save.
 >
+> **Symptom:** See per-fix sections below.
 >
 > **Status:** fixed on 2026-05-02 by uploading `model-auxiliary.safetensors` and
 > patching `model.safetensors.index.json`. The repo now carries an
 <!-- aux-rescue-banner: end -->
+<!-- aux-rescue-banner: begin -->
+> ## ⚠️ Update 2026-05-02 — please re-pull this model
+>
+> An earlier upload of this repo had its MoE expert weights in
+> per-expert (unfused) form: separate `experts.{N}.gate_proj.weight`,
+> `up_proj.weight`, `down_proj.weight` instead of the fused 3D
+> `experts.gate_up_proj` / `experts.down_proj` tensors the loader
+> expects. PEFT/LoRA wrapping during abliteration converts fused
+> 3D tensors into per-expert Linear modules; `merge_and_unload` +
+> `save_pretrained` then writes them in that wrong layout.
+>
+> **Symptom:** See per-fix sections below.
+>
+> **Status:** fixed on 2026-05-02 by re-fusing the per-expert tensors
+> back into 3D form (40 layers, 80 fused tensors)
+> and patching `model.safetensors.index.json`. The repo now carries
+> a `.moe_fuse.json` marker file so this fact is programmatically
+> verifiable. The original per-expert keys remain in the main shards
+> as harmless 'unexpected' keys at load time.
+>
+> Source used for shape reference: [`Qwen/Qwen3.5-35B-A3B`](https://huggingface.co/Qwen/Qwen3.5-35B-A3B)
+> (heretic's abliteration is preserved — fusion uses the dest's
+> per-expert weights, not source's).
+>
+> If you cloned this repo before 2026-05-02, please pull again.
+> Repaired with [`moe-fuse`](https://github.com/timrohrbaugh/aux-rescue).
+<!-- aux-rescue-banner: end -->
 # This is a decensored version of [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B), made using [Heretic](https://github.com/p-e-w/heretic) v1.2.0
 ## Abliteration parameters
 -----
 # Qwen3.5-35B-A3B
 <img width="400px" src="https://qianwen-res.oss-accelerate.aliyuncs.com/logo_qwen3.5.png">
 - **Next-Generation Training Infrastructure**: Near-100% multimodal training efficiency compared to text-only training and asynchronous RL frameworks supporting massive-scale agent scaffolds and environment orchestration.
 ![Benchmark Results](https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3.5/Figures/qwen3.5_middle_size_score.png)
 For more details, please refer to our blog post [Qwen3.5](https://qwen.ai/blog?id=qwen3.5).
 ## Model Overview
 - Type: Causal Language Model with Vision Encoder
     - MTP: trained with multi-steps
 - Context Length: 262,144 natively and extensible up to 1,010,000 tokens.
 ## Benchmark Results
 ### Language
 > Qwen3.5 models operate in thinking mode by default, generating thinking content signified by `<think>\n...</think>\n\n` before producing the final responses.
 > To disable thinking content and obtain direct response, refer to the examples [here](#instruct-or-non-thinking-mode).
 For streamlined integration, we recommend using Qwen3.5 via APIs. Below is a guide to use Qwen3.5 via OpenAI-compatible API.
 ### Serving Qwen3.5
 Qwen3.5 can be served via APIs with popular inference frameworks.
 In the following, we show example commands to launch OpenAI-Compatible API servers for Qwen3.5 models.
 > [!Important]
 > Inference efficiency and throughput vary significantly across frameworks.
 > We recommend using the latest framework versions to ensure optimal performance and compatibility.
 print("Chat response:", chat_response)
 ```
 #### Image Input
 ```python
 > [!Note]
 > If you are using APIs from Alibaba Cloud Model Studio, in addition to changing `model`, please use `"enable_thinking": False` instead of `"chat_template_kwargs": {"enable_thinking": False}`.
 ## Agentic Usage
 Qwen3.5 excels in tool calling capabilities.
 ### Qwen Code
 [Qwen Code](https://github.com/QwenLM/qwen-code) is an open-source AI agent for the terminal, optimized for Qwen models. It helps you understand large codebases, automate tedious work, and ship faster.
 For more information, please refer to [Qwen Code](https://qwenlm.github.io/qwen-code-docs/).
     Alternatively, override the default values via engine startup parameters. For implementation details, refer to: [vLLM](https://github.com/vllm-project/vllm/pull/34330) / [SGLang](https://github.com/sgl-project/sglang/pull/18467).
 ### Citation
 If you find our work helpful, feel free to give us a cite.