groxaxo/Qwen3.5-122B-A10B-abliterix-UD-Q4_K_XL-GGUF

Qwen3.5-122B-A10B-abliterix UD-Q4_K_XL GGUF

User-Defined (UD) mixed-precision quantization of Qwen3.5-122B-A10B-abliterix for llama.cpp.

What is UD (User-Defined) Quantization?

UD quantization is a per-tensor mixed-precision approach that goes beyond blanket quantization methods (like Q4_K_M or Q8_0 applied uniformly). Instead of using the same quantization type for every tensor, UD assigns an individually chosen type to each tensor based on its role in the model architecture and its sensitivity to precision loss.

How it differs from standard quantization

Aspect	Standard (e.g. Q4_K_M)	UD Quantization
Tensor types	One type for all (or a fixed pattern)	Each tensor gets its own type
Precision allocation	Uniform	Adaptive per tensor role
Importance matrix	Optional (general guidance)	Used to optimize k-quant block selection
Expert tensors	Same type as all others	Can be individually tuned

The UD methodology for this model

1. Architecture analysis: Qwen3.5-122B-A10B is a Mixture-of-Experts (MoE) model with 48 layers, 256 experts per layer (8 active), shared expert FFN, and SSM (state-space model) components. Each tensor was classified by its function:

   • Attention weights (attn_qkv, attn_gate, attn_q, attn_k, attn_v, attn_output): Quantized to Q8_0 — attention projections are highly sensitive to precision loss and benefit from near-lossless compression.

   • Expert FFN up/gate weights (ffn_up_exps, ffn_gate_exps): Quantized to Q4_K — these are the largest tensors in the model (256 experts x 3072x1024 each) and dominate the model size. Q4_K provides an excellent size/quality tradeoff for activation functions.

   • Expert FFN down weights (ffn_down_exps): Quantized to Q5_K — the down-projection is slightly more sensitive than up/gate, so Q5_K preserves more information here.

   • Shared expert FFN (ffn_down_shexp, ffn_up_shexp, ffn_gate_shexp, ffn_gate_inp_shexp): Quantized to Q8_0 — the shared expert processes every token, making it critical for quality.

   • Router/gating (ffn_gate_inp, ffn_gate_inp_shexp): Kept at F32 — the router decides which experts to activate. Even small precision errors here can cause misrouting, severely degrading output quality.

   • SSM components (ssm_a, ssm_conv1d, ssm_dt.bias, ssm_norm): Kept at F32 or Q8_0 — these are small tensors that play a structural role in the hybrid attention/SSM layers.

   • Norms (attn_norm, post_attention_norm, output_norm): Kept at F32 — normalization layers are tiny but critical for numerical stability.

   • Token embeddings (token_embd.weight): Quantized to Q8_0.

   • Output projection (output.weight): Quantized to Q8_0.

2. Importance matrix: A partial importance matrix (imatrix) was computed on coding-focused calibration data using llama-imatrix. This guides the k-quant block selection within each tensor, prioritizing precision on the most activation-sensitive dimensions. The imatrix was computed at n_ctx=512 across multiple calibration chunks before the quantization run.

3. Quantization: llama-quantize from llama.cpp was used with --tensor-type-file to apply the per-tensor type map, combined with --imatrix for importance-weighted block selection within each quantized tensor.

Resulting size

Component	Size
This GGUF file	~72 GB
Original BF16 GGUF	~244 GB
Compression ratio	~3.4x

Usage with llama.cpp

llama-server -m Qwen3.5-122B-A10B-abliterix-UD-Q4_K_XL.gguf \
  -ngl 999 \
  -c 32768 \
  -fa on \
  --reasoning auto

Multi-GPU tensor split example (3x RTX 3090 + 2x RTX 3060)

llama-server -m Qwen3.5-122B-A10B-abliterix-UD-Q4_K_XL.gguf \
  -ngl 999 \
  -ts 24,24,12,12,24 \
  -c 32768 \
  -ctk q8_0 -ctv q8_0 \
  -fa on \
  --reasoning auto

Provided by

• Quantized by: groxaxo
• Base model: wangzhang/Qwen3.5-122B-A10B-abliterix
• Quantization engine: llama.cpp build 8736