Instructions to use dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4")
model = AutoModelForCausalLM.from_pretrained("dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4

SGLang

How to use dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4
```

GLM-4.7-Flash-REAP-23B-A3B-NVFP4

NVFP4 PTQ checkpoint of cerebras/GLM-4.7-Flash-REAP-23B-A3B, produced with NVIDIA TensorRT Model Optimizer (hf_ptq).

This is a community quantized derivative. Credit for the base weights belongs to Cerebras (REAP prune) and Z.ai (original GLM-4.7-Flash). Please cite / attribute the upstream models when using this checkpoint.

Lineage

zai-org/GLM-4.7-Flash
        │
        │  Cerebras REAP — uniform 25% expert prune
        │  (arxiv:2510.13999)
        ▼
cerebras/GLM-4.7-Flash-REAP-23B-A3B   (~23B-A3B MoE, BF16)
        │
        │  ModelOpt PTQ — NVFP4 on routed experts
        │  (attention / gates / shared experts / embeds kept high-prec)
        ▼
dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4   (this repo)

Field	Value
Immediate base	`cerebras/GLM-4.7-Flash-REAP-23B-A3B`
Upstream ancestor	`zai-org/GLM-4.7-Flash`
Quantization tool	NVIDIA ModelOpt (`hf_ptq`)
Format	NVFP4 (group size 16)
Calibration data	NVIDIA Nemotron SFT / science / math / coding / agentic mixes (ModelOpt default calib set)
License	MIT (inherits from GLM-4.7-Flash / REAP card)

Quantization recipe (what changed)

From hf_quant_config.json (quant_algo: NVFP4, group_size: 16):

Component	Format
Routed MoE experts	NVFP4 (E2M1 + FP8 scales, block size 16)
Attention (`self_attn*`)	BF16 (excluded)
Router / `mlp.gate`	BF16 (excluded)
Shared experts	BF16 (excluded)
First dense layer (`layers.0*`)	BF16 (excluded)
Embeddings + `lm_head`	BF16 (excluded)
KV cache	not quantized in this export (`kv_cache_quant_algo: null`)

Note: ModelOpt calibration initially enabled FP8 KV scales; the published export has KV cache quantization disabled (GB10 / Spark-friendly: serve with --kv-cache-dtype auto).

Intended use

Same as the REAP MoE base, with much lower memory footprint on Blackwell GPUs (e.g. DGX Spark GB10). NVFP4 weight kernels need Blackwell.

Serve with vLLM

Validated on DGX Spark with vllm/vllm-openai:nightly-aarch64:

vllm serve dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4 \
  --quantization modelopt_fp4 \
  --moe-backend marlin \
  --enforce-eager \
  --kv-cache-dtype auto \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.161 \
  --served-model-name GLM-4.7-Flash-REAP-23B-A3B-NVFP4

Notes:

--moe-backend marlin was required for this MoE NVFP4 checkpoint on the tested nightly.
--enforce-eager and --kv-cache-dtype auto match the GB10 smoke/bench config (FP8 KV disabled).

DGX Spark speedups (BF16 → NVFP4)

vLLM on GB10, max-model-len=4096, concurrency 1.

Metric	BF16	NVFP4	Change
Memory (GiB)	50.78	23.67	−53%
Prefill TTFT (ms)	240.6	123.3	−49% (2.0× faster)
Throughput (tok/s)	24.5	28.5	+16% (1.16×)

Files to ignore in this local folder

If uploading from a local checkout that still has working artifacts, do not upload:

config.json.pre_kv_patch
hf_quant_config.json.pre_kv_patch
.quant_summary.txt

Disclaimer

This checkpoint is provided as-is for experimentation. Quantization can change model quality; run your own evals before production use. Not affiliated with Cerebras, Z.ai, or NVIDIA beyond use of open ModelOpt tooling.

Downloads last month: 13

Safetensors

Model size

13B params

Tensor type

F32

BF16

F8_E4M3

Model tree for dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4

Base model

zai-org/GLM-4.7-Flash

Finetuned

cerebras/GLM-4.7-Flash-REAP-23B-A3B

Quantized

(24)

this model

Paper for dmnsh/GLM-4.7-Flash-REAP-23B-A3B-NVFP4

REAP the Experts: Why Pruning Prevails for One-Shot MoE compression

Paper • 2510.13999 • Published Oct 15, 2025 • 20