Instructions to use ala-la/Qwen3.6-35B-A3B-pipeline-mixed34 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ala-la/Qwen3.6-35B-A3B-pipeline-mixed34 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ala-la/Qwen3.6-35B-A3B-pipeline-mixed34")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ala-la/Qwen3.6-35B-A3B-pipeline-mixed34")
model = AutoModelForMultimodalLM.from_pretrained("ala-la/Qwen3.6-35B-A3B-pipeline-mixed34", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ala-la/Qwen3.6-35B-A3B-pipeline-mixed34 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ala-la/Qwen3.6-35B-A3B-pipeline-mixed34"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ala-la/Qwen3.6-35B-A3B-pipeline-mixed34",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ala-la/Qwen3.6-35B-A3B-pipeline-mixed34

SGLang

How to use ala-la/Qwen3.6-35B-A3B-pipeline-mixed34 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ala-la/Qwen3.6-35B-A3B-pipeline-mixed34" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ala-la/Qwen3.6-35B-A3B-pipeline-mixed34",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ala-la/Qwen3.6-35B-A3B-pipeline-mixed34" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ala-la/Qwen3.6-35B-A3B-pipeline-mixed34",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ala-la/Qwen3.6-35B-A3B-pipeline-mixed34 with Docker Model Runner:
```
docker model run hf.co/ala-la/Qwen3.6-35B-A3B-pipeline-mixed34
```

Qwen3.6-35B-A3B — Mixed-Precision Quantization (AWQ + QuaRot)

Summary

Compressed Qwen3.6-35B-A3B-abliterated-v2 (uncensored MoE, 70.2 GB FP16) to 19.94 GB using a 4-stage pipeline: AWQ + QuaRot + mixed 3/4-bit quantization + bias compensation.

PPL: +0.03% (28.377 → 28.387) measured in-memory on 50 calibration texts (~682 tokens).

Metric	Value
FP16 PPL	28.377
Compressed PPL	28.387
Delta	+0.03%
FP16 size	70.2 GB
Compressed size	19.94 GB
Compression ratio	3.5x

⚠️ Important — Loading Limitations

This model uses custom QuantizedLinear layers not recognized by transformers.AutoModelForCausalLM. Loading via standard HF API produces random weights — the model will not work.

To use this model you need the pipeline code which defines QuantizedLinear and QuantizedMoeExperts classes. A proper load_quantized_model() function is not yet implemented — this is a research artifact, not a production-ready quantized model.

Not compatible with: llama.cpp, GGUF, vLLM, SGLang, Ollama, MLX, or any standard inference engine.

For production inference use lmstudio-community/Qwen3.6-35B-A3B-GGUF (Q4_K_M = 21.2 GB, works everywhere).

Architecture

Base: Qwen3.6-35B-A3B (MoE, 256 experts, top-8 routing, 3B active params)
Layers: 40 (30 GatedDeltaNet linear attention + 10 full attention)
Hidden dim: 2048
MoE experts: mixed 3-bit (40%) / 4-bit (60%), group_size=64
Attention/embed/lm_head: 8-bit
Vision encoder: not quantized (preserved from base)

Pipeline

Stage 1: AWQ (Activation-aware Weight Quantization)

Scales salient weight channels based on activation magnitudes. Scales fused into RMSNorm weights.

Paper: AWQ: Activation-aware Weight Quantization — Lin et al., 2023

Stage 2: QuaRot (Hadamard Rotation)

Rotates weight matrices with random Hadamard matrix to remove outlier features. RMSNorm weights fused into adjacent linear layers before rotation (required for rotation equivariance). Final RMSNorm fused into lm_head.

Paper: QuaRot: Outlier-free 4-bit Quantization — Ashkboos et al., 2024
Related: QuIP#: Hadamard Incoherence — Chee et al., 2023

Stage 3: Mixed-Precision Quantization

Per-expert sensitivity analysis → 3-bit for less sensitive experts, 4-bit for sensitive ones. Group quantization (group_size=64) with scale + zero-point.

Paper: GPTQ: Post-Training Quantization — Frantar et al., 2022
Paper: LLM.int8() — Dettmers et al., 2022

Stage 4: Bias Compensation

Corrects systematic quantization bias by comparing FP16 vs quantized layer outputs on calibration data.

Paper: Quantization Error Compensation — Li et al., 2023

Key Bug Fixes

7 bugs found and fixed during development:

#	Bug	Effect	Fix
1	GatedDeltaNet wrong projection names	30/40 layers unrotated	Fixed: in_proj_qkv/z/b/a + out_proj
2	MoE gate/router not rotated	Routing broken	Added gate + shared_expert_gate rotation
3	`--skip-deltanet-rotation` default=True	30/40 layers skipped	Default=False
4	Per-layer hooks caused double rotation	Residual stream rotated N times	Removed hooks, rotate embed+lm_head instead
5	AWQ gate not scaled	Routing broken	Added gate + shared_expert_gate scaling
6	RMSNorm fusion used `w` instead of `(1+w)`	x2 multiplier instead of (1+w)	Multiply by (1+w), set weight to zeros
7	Final RMSNorm not fused into lm_head	+47% PPL (norm not rotation-equivariant)	Fuse final norm weight into lm_head before rotation

Key insight: Qwen3.6 RMSNorm uses output = x_norm * (1 + weight), not x_norm * weight. Weight init=zeros, effective multiplier = 1+w. Fusion must multiply by (1+w) and set weight to zeros after.

Run History

Run	Config	PPL	Delta	Size
run1	QuaRot only (pre-fix)	71893	+253246%	17.9 GB
run2	QuaRot + 3/4-bit (pre-norm-fix)	45.8	+61.5%	17.9 GB
run3	QuaRot + 3/4-bit (all fixes)	30.89	+8.87%	17.9 GB
run4	AWQ + QuaRot + bias (30% 4-bit)	30.04	+5.86%	17.9 GB
run5	AWQ + QuaRot + bias (50% 4-bit)	29.64	+4.45%	18.5 GB
run6	All 4-bit, gs=64	28.23	-0.53%	21.2 GB
run7	50% 4-bit, gs=64	28.98	+2.12%	19.6 GB
run8	60% 4-bit, gs=64	28.39	+0.03%	19.94 GB

Evaluation

Test	Status	Notes
PPL (50 samples, 682 tokens)	✓	+0.03% vs FP16 (in-memory)
QuaRot rotation lossless	✓	-0.11% without quantization
HumanEval	✗	Not run
MMLU	✗	Not run
HellaSwag	✗	Not run
ARC	✗	Not run
GSM8K	✗	Not run
Long context	✗	Not tested

Limitations:

PPL measured in-memory before saving — not verified after save/load cycle
No standard benchmarks run
Custom format — model does not load via standard HF API
+0.03% on 682 tokens is within noise — real degradation unknown
load_quantized_model() function not implemented

Files

model-*.safetensors — quantized weights (custom format)
config.json — model config (quantization_config: custom)
quantization_meta.json — per-layer quantization metadata
zero_loss_pipeline_cuda.py — pipeline code (QuantizedLinear, QuantizedMoeExperts)
METHODS_LOG.md — full development history
RESULTS.md — run results table

References

AWQ — Lin et al., 2023
QuaRot — Ashkboos et al., 2024
QuIP# — Chee et al., 2023
GPTQ — Frantar et al., 2022
LLM.int8() — Dettmers et al., 2022
Quantization Error Compensation — Li et al., 2023

Base Model

wangzhang/Qwen3.6-35B-A3B-abliterated-v2 — uncensored abliterated version
Qwen/Qwen3.6-35B-A3B — original

License

Tongyi Qianwen (same as base model).

Downloads last month: 643

Safetensors

Model size

9B params

Tensor type

I64

I32

F16

Model tree for ala-la/Qwen3.6-35B-A3B-pipeline-mixed34

Base model

Qwen/Qwen3.6-35B-A3B

Finetuned

wangzhang/Qwen3.6-35B-A3B-abliterated-v2

Quantized

(10)

this model

Papers for ala-la/Qwen3.6-35B-A3B-pipeline-mixed34

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

Paper • 2210.17323 • Published Oct 31, 2022 • 11