Fredred89/Qwen3.5-27B-GLM5.1-Distill-v1-GGUF

Qwen3.5-27B-GLM5.1-Distill-v1 — APEX Quantized GGUF

Architecture: DENSE (NOT MoE) — All 27B parameters are active on every token. No expert routing.

Model Info

Property	Value
Base model	Qwen3.5-27B-GLM5.1-Distill-v1
Parameters	27B (all active, dense)
Layers	64 (48 GDN recurrent + 16 full-attention, every 4th layer)
Architecture	Qwen3_5ForConditionalGeneration (hybrid GDN)
Vocab	248,320
Context	262,144

Available Quantizations

File	Type	Size	PPL	Notes
...-APEX-Quality-v5.gguf	APEX Q4_K_M + edge upgrades	16.18 GB	5.5596	Best quality — beats built-in Q4_K_M
...-Q4_K_M.gguf	Built-in Q4_K_M	15.41 GB	5.5687	Baseline
...-Q8_0.gguf	Q8_0	~27 GB	—	High quality reference
...-F16-fixed2.gguf	F16	53.8 GB	~5.55	Full precision (fixed block_count)

Benchmark: wiki.test.raw, c=2048, chunks=10

APEX Quality v5 — Method

This uses an APEX-inspired minimal-override strategy adapted for dense models.

What APEX Quality v5 does:

• 93 tensor-type-file overrides — only edge layer upgrades
• Edge layers L0-7, L56-63: q4_K → q5_K
• token_embd.weight: q4_K → q6_K
• Everything else: no override (built-in k-quant mixture handles it)

Why this works for dense models:

• Unlike MoE models (where 97% of expert params are inactive per token), dense models have all parameters active on every forward pass
• Built-in llama.cpp k-quant mixture is already near-optimal for dense models
• Only edge layers benefit from upgrades (embedding alignment + logit generation)
• Full-replacement APEX tiers perform worse on this dense model than minimal overrides

Key difference from APEX paper: The original APEX paper targets MoE models (Qwen3.5-35B-A3B with 256 routed experts). Its biggest innovation — compressing inactive experts aggressively — doesn't apply here. We only use the layer-gradient principle.

Reproduction

# 1. Convert to F16 and fix metadata (block_count bug: 65 → 64)
python3 convert_hf_to_gguf.py safetensors_source/ --outfile model-F16.gguf --outtype f16
llama-quantize \
  --override-kv 'qwen35.block_count=int:64' \
  --override-kv 'qwen35.nextn_predict_layers=int:0' \
  model-F16.gguf model-F16-fixed.gguf COPY

# 2. Quantize with APEX tensor-type-file (see APEX-Quality-v5.tensor_types.txt)
llama-quantize \
  --tensor-type-file APEX-Quality-v5.tensor_types.txt \
  model-F16-fixed.gguf APEX-Quality-v5.gguf Q4_K_M

# 3. Benchmark (MUST use c=2048, not default c=512)
llama-perplexity -m APEX-Quality-v5.gguf -f wiki.test.raw -c 2048 --chunks 10 -t 4

Credits

• APEX methodology: LocalAI/apex-quant — Ettore Di Giacinto, Richard Palethorpe
• Base model: Jackrong/Qwen3.5-27B-GLM5.1-Distill-v1
• Quantization: llama.cpp stock tooling, no custom kernels