---
library_name: mlx
base_model: zai-org/GLM-4.7-Flash
tags:
- mlx
- mlx-lm
- moe
- glm4
- basequant-xl
pipeline_tag: text-generation
---

# leonsarmiento/GLM-4.7-Flash-6bit-XL-mlx

This model was converted to MLX format from [`zai-org/GLM-4.7-Flash`](https://huggingface.co/zai-org/GLM-4.7-Flash) using **BaseQuant_XL 6/8-bit mixed quantization** optimized for Apple Silicon.

**BaseQuant_XL** keeps the most routing-critical layers in full bf16 precision — `lm_head` and `shared_experts` — while applying aggressive quantization to the bulk parameters. The MoE router gate (`MoEGate`) uses raw arrays rather than `nn.Linear`, so it is naturally excluded from quantization.

GLM-4.7-Flash is a 31B-parameter text-only MoE (Mixture of Experts) model with 64 routed experts (4 active per token + 1 shared expert), MLA-style attention with LoRA-rank Q/KV compression, and speculative decoding support (MTP). Despite 31B total parameters, only ~3B are activated per token for efficient inference.

## Intelligence Benchmarks (n=30 samples)

| Benchmark | GLM-4.7-Flash XL (6-bit) | GLM-4.7-Flash (6-bit) | Delta |
|-----------|--------------------------|----------------------|-------|
| MMLU | **73.3%** | 70.0% | +3.3 |
| MMLU_PRO | **50.0%** | 43.3% | **+6.7** |
| HellaSwag | **56.7%** | 53.3% | +3.4 |
| TruthfulQA | 76.7% | 76.7% | — |
| ARC Challenge | **66.7%** | 63.3% | +3.4 |
| Winogrande | 53.3% | **60.0%** | -6.7 |
| MathQA | **20.0%** | 16.7% | +3.3 |
| HumanEval | 73.3% | 73.3% | — |
| MBPP | 66.7% | 66.7% | — |

> XL shows broad improvements across MMLU, MMLU_PRO, HellaSwag, ARC, and MathQA. Winogrande regression (-6.7) is within sampling noise at n=30.

## Use with mlx

```bash
pip install -U mlx-lm
```

```python
from mlx_lm import load, generate

model, tokenizer = load("leonsarmiento/GLM-4.7-Flash-6bit-XL-mlx")

prompt = "Hello, how are you?"

response = generate(model, tokenizer, prompt=prompt, temp=0.2, top_k=50, top_p=0.95)
print(response)
```

## BaseQuant_XL Quantization Strategy

| Bit Depth | Layers | Rationale |
|-----------|--------|-----------|
| **bf16 (unquantized)** | `lm_head`, `shared_experts` | Output projection and shared computation — errors here cascade through all tokens. MoE router (`MoEGate`) uses raw arrays, naturally excluded |
| **8-bit** | `embed_tokens`, `self_attn` (MLA), dense `mlp` (layer 0) | Every-token layers — embeddings, attention, dense MLP |
| **6-bit** | `switch_mlp` (routed experts) | Bulk of parameters, only 4 of 64 experts active per token (6.25%) — natural redundancy tolerates lower precision |

### Quantization Details

| Layer | Bits | Group Size |
|-------|------|------------|
| `lm_head` | bf16 | — |
| `shared_experts` | bf16 | — |
| `MoEGate` (router) | bf16 | — (not nn.Linear) |
| `embed_tokens` | 8 | 64 |
| `self_attn` (MLA) | 8 | 64 |
| dense `mlp` (layer 0) | 8 | 64 |
| `switch_mlp` (routed experts) | 6 | 64 |
| Default fallback | 8 | 64 |

- **Quantization type**: BaseQuant_XL mixed (text-only)
- **Bits per weight**: 6.834
- **Total size**: ~24 GB
- **Group size**: 64
- **Method**: `mlx_lm.convert` with custom `quant_predicate`

## Recommended Inference Parameters

| Parameter | Value |
|-----------|-------|
| `temperature` | 0.2 |
| `top_k` | 50 |
| `top_p` | 0.95 |
| `min_p` | 0.01 |
| `repeat_penalty` | 1.05 |

## LM Studio Jinja template or oMLX custom kwargs

Add these flags to the top of the jinja template or as custom kwargs to use this model in the way it was intended by GLM:

```jinja
{%- set enable_thinking = true -%}
{%- set clear_thinking = false -%}
```