leonsarmiento/GLM-4.7-Flash-6bit-XL-mlx

This model was converted to MLX format from zai-org/GLM-4.7-Flash using BaseQuant_XL 6/8-bit mixed quantization optimized for Apple Silicon.

BaseQuant_XL keeps the most routing-critical layers in full bf16 precision — lm_head and shared_experts — while applying aggressive quantization to the bulk parameters. The MoE router gate (MoEGate) uses raw arrays rather than nn.Linear, so it is naturally excluded from quantization.

GLM-4.7-Flash is a 31B-parameter text-only MoE (Mixture of Experts) model with 64 routed experts (4 active per token + 1 shared expert), MLA-style attention with LoRA-rank Q/KV compression, and speculative decoding support (MTP). Despite 31B total parameters, only ~3B are activated per token for efficient inference.

Intelligence Benchmarks (n=30 samples)

Benchmark	GLM-4.7-Flash XL (6-bit)	GLM-4.7-Flash (6-bit)	Delta
MMLU	73.3%	70.0%	+3.3
MMLU_PRO	50.0%	43.3%	+6.7
HellaSwag	56.7%	53.3%	+3.4
TruthfulQA	76.7%	76.7%	—
ARC Challenge	66.7%	63.3%	+3.4
Winogrande	53.3%	60.0%	-6.7
MathQA	20.0%	16.7%	+3.3
HumanEval	73.3%	73.3%	—
MBPP	66.7%	66.7%	—

XL shows broad improvements across MMLU, MMLU_PRO, HellaSwag, ARC, and MathQA. Winogrande regression (-6.7) is within sampling noise at n=30.

Use with mlx

pip install -U mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("leonsarmiento/GLM-4.7-Flash-6bit-XL-mlx")

prompt = "Hello, how are you?"

response = generate(model, tokenizer, prompt=prompt, temp=0.2, top_k=50, top_p=0.95)
print(response)

BaseQuant_XL Quantization Strategy

Bit Depth	Layers	Rationale
bf16 (unquantized)	lm_head, shared_experts	Output projection and shared computation — errors here cascade through all tokens. MoE router (MoEGate) uses raw arrays, naturally excluded
8-bit	embed_tokens, self_attn (MLA), dense mlp (layer 0)	Every-token layers — embeddings, attention, dense MLP
6-bit	switch_mlp (routed experts)	Bulk of parameters, only 4 of 64 experts active per token (6.25%) — natural redundancy tolerates lower precision

Quantization Details

Layer	Bits	Group Size
lm_head	bf16	—
shared_experts	bf16	—
MoEGate (router)	bf16	— (not nn.Linear)
embed_tokens	8	64
self_attn (MLA)	8	64
dense mlp (layer 0)	8	64
switch_mlp (routed experts)	6	64
Default fallback	8	64

• Quantization type: BaseQuant_XL mixed (text-only)
• Bits per weight: 6.834
• Total size: ~24 GB
• Group size: 64
• Method: mlx_lm.convert with custom quant_predicate

Recommended Inference Parameters

Parameter	Value
temperature	0.2
top_k	50
top_p	0.95
min_p	0.01
repeat_penalty	1.05

LM Studio Jinja template or oMLX custom kwargs

Add these flags to the top of the jinja template or as custom kwargs to use this model in the way it was intended by GLM:

{%- set enable_thinking = true -%}
{%- set clear_thinking = false -%}