Instructions to use brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw")
model = AutoModelForCausalLM.from_pretrained("brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Trellis

How to use brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw with Trellis:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw

SGLang

How to use brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw with Docker Model Runner:
```
docker model run hf.co/brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw
```

GLM-5.2 EXL3 TR3 3.0 bpw

This is a TP4, rank-sliced EXL3 build of zai-org/GLM-5.2, optimized for four NVIDIA Blackwell workstation GPUs. Routed MoE experts in layers 3-77 use EXL3 Trellis weights targeting 3.0 bits per weight. Accuracy-sensitive and dense components remain in BF16.

The repository payload is 332.19 GB (309.37 GiB). This format requires the custom vLLM + Sparkinfer runtime below; it is not a drop-in Transformers model. The config.json retains ModelOpt/NVFP4 compatibility metadata used by the conversion pipeline, but the routed weights are EXL3 and the required launch flag is --quantization exl3. NVFP4 in the supplied runtime refers to the KV cache, not the routed-expert weight format.

Quantization layout

Component	Storage
Routed MoE experts, layers 3-77	EXL3 Trellis, TP4 rank-sliced, 3.0 bpw target
Dense MLP layers 0-2	BF16
Shared-expert MLPs	BF16
Attention and sparse indexer	BF16
Embeddings and LM head	BF16
Norms, router gates, and e-score correction bias	BF16/FP32 source precision
MTP layer 78	BF16

The calibration manifest is included as calibration_manifest.json. It records 12,228 owner-corpus samples across general, legal, coding/agentic, and reasoning/termination axes. The model was calibrated at TP8 and packed for TP4.

Supported runtime

The tested runtime is published at verdictai/glm52-exl3-sparkinfer:

verdictai/glm52-exl3-sparkinfer:v31-gg-v20-sic3828fd-vllm0c79e41-cu132-sm120a

The supplied scripts pin the immutable registry manifest:

verdictai/glm52-exl3-sparkinfer:v31-gg-v20-sic3828fd-vllm0c79e41-cu132-sm120a@sha256:0433ae94665b769b78dd301f952d907508a3ba80bce47a1630ec20ade8812dff

It pins:

Gilded Gnosis v20 canonical vLLM 6722c1d (dev/gilded-gnosis) + EXL3 Trellis (rebased PR #139)
Sparkinfer v20 canonical 1a88b389 (master) + EXL3 Trellis fused arm (rebased PR #49)
CUDA 13.2, PyTorch 2.12, CUTLASS DSL 4.6.0, FlashInfer 801d57a, NCCL 2.30.4, SM120a
EXL3 Trellis MoE, B12X sparse MLA, DCP A2A, and MTP speculative decoding
NVFP4 DeepSeek-MLA KV cache using calibrated outer scales

v20 changes (Gilded Gnosis v20 base)

This release rebases the EXL3 Trellis backend onto the Gilded Gnosis v20 canonical heads (vLLM 6722c1d, Sparkinfer 1a88b389). The v20 base supplies the upstreamed MTP/DCP correctness fixes — head-major cross-rank BMM (vLLM #147), MTP verifier-decode dispatch (vLLM #164), and graph-resource isolation (vLLM #149). On top of that base this image adds (vLLM PR #139 / Sparkinfer PR #49, both rebased onto v20):

EXL3 Trellis MoE backend for rank-sliced GLM/DeepSeek routed experts.
MTP tool-call + DSA-crash fix — the structured-output grammar advances from the authoritative step delta so tool calling engages under speculative decoding, and has_indexer is derived from index_k so MTP draft steps that skip top-k no longer trip the fused-norm-rope assertion.
Dual-plan Trellis prefill — prefill batches route through the planned Trellis MoE.
SM120 + B12X flattening fix — MTP next_n>2 uses the native (B, next_n) sparse-indexer path instead of the DeepGEMM next_n<=2 flatten fallback, which had corrupted MTP-3 code generation.

Validated on 4x RTX PRO 6000 Blackwell 96 GB (TP4/DCP4, MTP-3 greedy, NVFP4 KV, FULL_AND_PIECEWISE cudagraphs):

Code generation — the previously reported ~50% syntax-error rate under MTP-3 is eliminated; generated Python/HTML is 94–100% syntactically valid across runs. The rare remaining edge is fp8-KV/DCP floating-point nondeterminism, not the prior systematic corruption.
LAVD long-context retrieval (r10/c5, temp 0): 10/10 (6 exact, 4 near, 0 fail) — up from 8/10 (2 fails) on the prior base. The v20 canonical MTP fixes plus the flattening fix drive the fail count from 2 to 0.

Asynchronous scheduling remains disabled as a correctness guard for this DCP4/MTP path; do not enable it on this release.

Quick start

Install Docker Engine, Docker Compose v2, the NVIDIA Container Toolkit, and the Hugging Face CLI. Then download the model and start the server:

hf download brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw \
  --local-dir "$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"

cd "$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"
chmod +x server.sh
./server.sh start
./server.sh logs

The OpenAI-compatible endpoint is available locally at http://localhost:8000/v1. Here, localhost always means the machine on which the downloader starts this model; it does not refer to the model publisher's machine.

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.2-EXL3-TR3-3.0bpw",
    "messages": [{"role": "user", "content": "What is 2 + 2?"}],
    "temperature": 0,
    "max_tokens": 128
  }'

Useful controls:

./server.sh status
./server.sh logs
./server.sh restart
./server.sh stop

Direct Compose use

server.sh exports the defaults and invokes the included docker-compose.yml. To run Compose directly, set the model and cache paths:

export MODEL_DIR="$HOME/models/GLM-5.2-EXL3-TR3-3.0bpw"
export CACHE_DIR="$HOME/.cache/glm52-exl3-sparkinfer"
docker compose pull
docker compose up -d
docker compose logs --tail 100 -f glm52

The defaults can be overridden without editing the files:

Variable	Default	Purpose
`MODEL_DIR`	Directory containing `server.sh`	Model mount
`CACHE_DIR`	`~/.cache/glm52-exl3-sparkinfer`	Persistent JIT cache
`PORT`	`8000`	Host API port
`BIND_ADDRESS`	`127.0.0.1`	Local-only host binding
`CUDA_VISIBLE_DEVICES`	`3,1,2,0`	Physical GPU to TP-rank order
`GPU_MEMORY_UTILIZATION`	`0.96`	vLLM memory reservation
`MAX_MODEL_LEN`	`262144`	Per-request context cap
`MTP_TOKENS`	`3`	Validated speculative-token count (MTP-3)
`ENABLE_ASYNC_SCHEDULING`	`0`	Required correctness guard
`NUM_GPU_BLOCKS_OVERRIDE`	`1024`	262,144 logical KV tokens at DCP4

The tested GPU order intentionally keeps physical GPU 3 away from TP rank 3. On another host, set CUDA_VISIBLE_DEVICES=0,1,2,3 or use the order appropriate for that machine.

The supplied Compose file binds only to loopback by default, so it does not publish the API to the LAN or internet.

Runtime validation

The exact published image completed all 81 model-shard loads, EXL3 initialization, Sparkinfer PCIe collective initialization, NVFP4 KV allocation, and full plus piecewise CUDA graph capture. The v20 base has a larger runtime footprint, so the release preset allocates 262,144 logical KV-cache tokens: 1,024 blocks x 64 tokens x DCP4, with 65,536 tokens local to each DCP rank. The configured per-request context cap is also 262,144. A GPU 0 with no other resident processes supports a larger KV pool and context (raise NUM_GPU_BLOCKS_OVERRIDE / MAX_MODEL_LEN within the KV capacity reported at startup).

Quality was validated with the release serving configuration (MTP-3, TP4/DCP4, concurrency 5):

Evaluation	v20 result	Prior base
LAVD r10/c5	10/10 (6 exact, 4 near, 0 fail)	8/10 (2 fail)
Code generation (Python/HTML, temp 0)	94–100% valid	~50% (reported bug)

The LAVD fail count dropping 2 → 0 reflects the v20 canonical MTP fixes plus the SM120+B12X flattening fix. Exact/near split varies run to run (fp8-KV/DCP nondeterminism); the 0-fail result is the stable signal.

The following prefill, decode, and KLD tables are indicative reference measured on the prior base; the v20 image is validated for correctness by the results above.

Cold standalone prefill results:

Requested context	Prompt tokens	TTFT	Client tok/s	Server tok/s
8K	8,201	5.46 s	1,502	1,507
64K	64,512	51.64 s	1,249	1,252
128K	128,881	109.00 s	1,182	1,185

Sustained decode used 20-second steady-state cells after warmup, zero input context, and continuous OpenAI stream usage:

Concurrency	1	2	3	4	5	6	7	8
Aggregate tok/s	48.9	112.9	154.0	188.2	218.2	239.4	253.9	266.8
Per-request tok/s	48.9	56.4	51.3	47.0	43.6	39.9	36.3	33.3

A separate 30-second C1 run measured 48.5 tok/s. No decode cell was underfilled, capacity-limited, or errored.

Five-run, 2,047-position DCP4 KLD against the same verified BF16 reference:

KV cache	Mean KLD	Sample SD	Min	Max
NVFP4 DeepSeek MLA	0.1124021	0.0025948	0.1086084	0.1156108
FP8	0.1036666	0.0018374	0.1016535	0.1066077

Full methodology, raw JSON, and copyable Rich TUI logs are in benchmarks/2026-07-22.

Source

License

The model and this derivative are released under the MIT license. See LICENSE and the upstream model card for attribution and usage terms.

Downloads last month: 651

Safetensors

Model size

158B params

Tensor type

BF16

F32

F16

I16

Model tree for brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw

Base model

zai-org/GLM-5.2

Quantized

(124)

this model

Paper for brandonmusic/GLM-5.2-EXL3-TR3-3.0bpw

GLM-5: from Vibe Coding to Agentic Engineering

Paper • 2602.15763 • Published Feb 17 • 205