upload checkpoint

README.md

@@ -0,0 +1,136 @@
+---
+language:
+- en
+library_name: transformers
+tags:
+- glm
+- MOE
+- pruning
+- compression
+license: mit
+name: cerebras/GLM-4.6-REAP-268B-A32B
+description: >
+  This model was obtained by uniformly pruning 25% of experts in GLM-4.6 using the REAP method.
+readme: >
+  https://huggingface.co/cerebras/GLM-4.6-REAP-268B-A32B/main/README.md
+license_link: https://huggingface.co/zai-org/GLM-4.6/blob/main/LICENSE
+pipeline_tag: text-generation
+base_model:
+- zai-org/GLM-4.6
+---
+
+<p align="center">
+  <em>𓌳 <strong>REAP</strong>𓌳  the Experts: Why Pruning Prevails for One-Shot MoE Compression</em><br>
+  <img src="https://i.imgur.com/rmzG3gg.png" alt="REAP" width="75%">
+</p>
+
+# GLM-4.6-REAP-268B-A32B
+
+## ✨ Highlights
+
+Introducing **GLM-4.6-REAP-268B-A32B**, a **memory-efficient compressed variant** of GLM-4.6 that maintains near-identical performance while being **25% lighter**.
+
+**Note: this is a BF16 version for more accurate downstream low-bit quantization. An [FP8 version](https://huggingface.co/cerebras/GLM-4.6-REAP-268B-A32B-FP8) is also available on HF.**
+
+This model was created using **REAP (Router-weighted Expert Activation Pruning)**, a novel expert pruning method that selectively removes redundant experts while preserving the router's independent control over remaining experts. Key features include:
+
+- **Near-Lossless Performance**: Maintains almost identical accuracy on code generation, agentic coding, and function calling tasks compared to the full 355B model
+- **25% Memory Reduction**: Compressed from 355B to 268B parameters, significantly lowering deployment costs and memory requirements
+- **Preserved Capabilities**: Retains all core functionalities including code generation, agentic workflows, repository-scale understanding, and function calling
+- **Drop-in Compatibility**: Works with vanilla vLLM - no source modifications or custom patches required
+- **Optimized for Real-World Use**: Particularly effective for resource-constrained environments, local deployments, and academic research
+---
+## 📋 Model Overview
+
+**GLM-4.6-REAP-268B-A32B** has the following specifications:
+
+- **Base Model**: GLM-4.6
+- **Compression Method**: REAP (Router-weighted Expert Activation Pruning)
+- **Compression Ratio**: 25% expert pruning
+- **Type**: Sparse Mixture-of-Experts (SMoE) Causal Language Model
+- **Number of Parameters**: 268B total, 32B activated per token
+- **Number of Layers**: 92
+- **Number of Attention Heads (GQA)**: 96 for Q and 8 for KV
+- **Number of Experts**: 120 (uniformly pruned from 160)
+- **Number of Activated Experts**: 8 per token
+- **Context Length**: 202,752 tokens
+- **License**: MIT
+
+---
+
+## 📊 Evaluations
+
+TBD for BF16 model. [Evalulation results available for the FP8 variant](https://huggingface.co/cerebras/GLM-4.6-REAP-268B-A32B-FP8#%F0%9F%93%8A-evaluations).
+
+For more details on the evaluation setup, refer to the [REAP arXiv preprint](https://arxiv.org/abs/2510.13999).
+
+---
+
+## 🚀 Deployment
+
+You can deploy the model directly using the **latest vLLM** (v0.11.0), no source modifications or custom patches required.
+
+```bash
+vllm serve cerebras/GLM-4.6-REAP-268B-A32B \
+    --tensor-parallel-size 8 \
+    --tool-call-parser glm45 \
+    --enable-auto-tool-choice \
+    --enable-expert-parallel
+```
+
+If you encounter insufficient memory when running this model, you might need to set a lower value for `--max-num-seqs` flag (e.g. set to 64).
+
+
+## 🧩 Model Creation
+
+This checkpoint was created by applying the **REAP (Router-weighted Expert Activation Pruning)** method uniformly across all Mixture-of-Experts (MoE) blocks of **GLM-4.6**, with a **25% pruning rate**.
+
+### How REAP Works
+
+REAP selects experts to prune based on a novel **saliency criterion** that considers both:
+- **Router gate values**: How frequently and strongly the router activates each expert
+- **Expert activation norms**: The magnitude of each expert's output contributions
+
+This dual consideration ensures that experts contributing minimally to the layer's output are pruned, while preserving those that play critical roles in the model's computations.
+
+### Key Advantages
+
+- **One-Shot Compression**: No fine-tuning required after pruning - the model is immediately ready for deployment
+- **Preserved Router Control**: Unlike expert merging methods, REAP maintains the router's independent, input-dependent control over remaining experts, avoiding "functional subspace collapse"
+- **Generative Task Superiority**: REAP significantly outperforms expert merging approaches on generative benchmarks (code generation, creative writing, mathematical reasoning) while maintaining competitive performance on discriminative tasks
+
+### Calibration
+
+The model was calibrated using a diverse mixture of domain-specific datasets including:
+- Code generation samples ([evol-codealpaca](https://huggingface.co/datasets/theblackcat102/evol-codealpaca-v1))
+- Function calling examples ([xlam-function-calling](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k))
+- Agentic multi-turn trajectories ([SWE-smith-trajectories](https://huggingface.co/datasets/SWE-bench/SWE-smith-trajectories))
+
+📚 For more details, refer to the following resources:
+
+- [🧾 arXiv Preprint](https://arxiv.org/abs/2510.13999)
+- [🧾 REAP Blog](https://www.cerebras.ai/blog/reap)
+- [💻 REAP Codebase (GitHub)](https://github.com/CerebrasResearch/reap)
+
+---
+
+## ⚖️ License
+
+This model is derived from
+**[`zai-org/GLM-4.6`](https://huggingface.co/zai-org/GLM-4.6)**
+and distributed under the **MIT license**.
+
+---
+
+## 🧾 Citation
+
+If you use this checkpoint, please cite the REAP paper:
+
+```bibtex
+@article{lasby-reap,
+  title={REAP the Experts: Why Pruning Prevails for One-Shot MoE compression},
+  author={Lasby, Mike and Lazarevich, Ivan and Sinnadurai, Nish and Lie, Sean and Ioannou, Yani and Thangarasa, Vithursan},
+  journal={arXiv preprint arXiv:2510.13999},
+  year={2025}
+}
+```

chat_template.jinja

@@ -0,0 +1,103 @@
+[gMASK]<sop>
+{%- if tools -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{{ tool | tojson(ensure_ascii=False) }}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}
+<arg_key>{arg-key-1}</arg_key>
+<arg_value>{arg-value-1}</arg_value>
+<arg_key>{arg-key-2}</arg_key>
+<arg_value>{arg-value-2}</arg_value>
+...
+</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) %}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {% set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{% for m in messages %}
+{%- if m.role == 'user' -%}<|user|>
+{{ visible_text(m.content) }}
+{{- '/nothink' if (enable_thinking is defined and not enable_thinking and not visible_text(m.content).endswith("/nothink")) else '' -}}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set reasoning_content = '' %}
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- else %}
+    {%- if '</think>' in content %}
+        {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+        {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+    {%- endif %}
+{%- endif %}
+{%- if loop.index0 > ns.last_user_index and reasoning_content -%}
+{{ '\n<think>' + reasoning_content.strip() +  '</think>'}}
+{%- else -%}
+{{ '\n<think></think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ '\n' + content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{ '\n<tool_call>' + tc.name }}
+{% set _args = tc.arguments %}
+{% for k, v in _args.items() %}
+<arg_key>{{ k }}</arg_key>
+<arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>
+{% endfor %}
+</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if m.content is string -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' }}
+{%- endif %}
+{{- '\n<tool_response>\n' }}
+{{- m.content }}
+{{- '\n</tool_response>' }}
+{%- else -%}
+<|observation|>{% for tr in m.content %}
+
+<tool_response>
+{{ tr.output if tr.output is defined else tr }}
+</tool_response>{% endfor -%}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>
+{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    <|assistant|>{{- '\n<think></think>' if (enable_thinking is defined and not enable_thinking) else '' -}}
+{%- endif -%}

config.json

@@ -0,0 +1,43 @@
+{
+  "architectures": [
+    "Glm4MoeForCausalLM"
+  ],
+  "attention_bias": true,
+  "attention_dropout": 0.0,
+  "eos_token_id": [
+    151329,
+    151336,
+    151338
+  ],
+  "first_k_dense_replace": 3,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 12288,
+  "max_position_embeddings": 202752,
+  "model_type": "glm4_moe",
+  "moe_intermediate_size": 1536,
+  "n_group": 1,
+  "n_routed_experts": 120,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 96,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 92,
+  "num_key_value_heads": 8,
+  "num_nextn_predict_layers": 0,
+  "pad_token_id": 151329,
+  "partial_rotary_factor": 0.5,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "routed_scaling_factor": 2.5,
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.55.0",
+  "use_cache": true,
+  "use_qk_norm": true,
+  "vocab_size": 151552
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": [
+    151329,
+    151336,
+    151338
+  ],
+  "pad_token_id": 151329,
+  "transformers_version": "4.55.0"
+}

model-00001-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0a6e5e9d0f366c4d25e9e213a2a22037eacfb25c6ae133b9d5aa09d5d8a7c008
+size 4986172552

model-00002-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8301e420fcbccfd4feaa1b549aaa9cbd604aaecf857f9578f9edcb2de9710fc9
+size 4992539416

model-00003-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5ac4ad6911327ec3c275045057623683d0faa3134a85ca5d4d2f8337e8be8152
+size 4986018208

model-00004-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe60d31fcf6cc6a4e387a4560559e5a24b90e681f77567587490e2ef3f6ff85f
+size 4992539368

model-00005-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ec9deac512c4c455f6df1550c700954f8d4a6b5cbf5aad0ae8d09f08535ce8f
+size 4992539400

model-00006-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f47ebd25e9e52215d679441718856ac91573a89bdabd25e57de4bfe6617d4a27
+size 4992539400

model-00007-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:752cd7ad100994ed75bec3380c1b86174c87877dd3855e9ded6c81e25823ab00
+size 4992539400

model-00008-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bad740f814efabbf0f3c4202d84ba23875d36f5242a80a6fbe930e012c93e6e2
+size 4992539416

model-00009-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9cb77a47059eb20e18b93166e2f5f23fd5f0470f1a87b2c28bd93942b227fb9e
+size 4986018208

model-00010-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:848efa0e6dcfe8e08a26c851b9d95e3249ad698610d81298d250c519413b3d58
+size 4992539656

model-00011-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc1ef2fd5e7543b7df4bb4ea45f063aca4beede4a7ab5e4c782c0542f0788b58
+size 4992539712

model-00012-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7104aa642080b2e901970c5af6704704cdc7a79c13c175c595a27ff909c74956
+size 4992539712

model-00013-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2ff6145a4893cb3b9d5b34684bbdae5885256878d42a772f7caa5a0785bf2923
+size 4992539712

model-00014-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0f9865f55e6f69dc4a6f1d58621566545c116c19eb2c56234d32fd38a6c74181
+size 4992539728

model-00015-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a4015a4dad29414424a3ec8894b0e1852b72476240bdea269d436184b071491c
+size 4986018528

model-00016-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d00feb767164b595de12f5a56d738d89499efbee3e29dc52e07abd5f461450c
+size 4992539672

model-00017-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:295a9eaf53e36ea514e60854665098e68ee7709e0aaccc8072b8d6abac9b565e
+size 4992539712

model-00018-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:95563abc097743b4566638916fbabb193a09b0bc902a8ba6658ed71fb12cb5a8
+size 4992539712

model-00019-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e0e8bedcdabc51ca6935e9c3a5d62bce83dd8b81c40e25d80655f54f29f0f58
+size 4992539712

model-00020-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:861fb0dc59ba06ffa6cd1417afcc4edc448b94267d1c1d57174109b9ccfc13e3
+size 4992539720

model-00021-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c418811dcc87cd9a5c5cdb8e25c1bd69c9b18d9e122558433b63f9d23e2805c1
+size 4986018536

model-00022-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:059eb68d94e792c72596e77c5bae5f72c74db124c97e3c529809f86ded379b68
+size 4992539672

model-00023-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fd3d78753eb6b4bcd54cd2f79e823e1df145189d13e295e0cd0c88f62e3cefd0
+size 4992539712

model-00024-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:723dca4b658f293db283aa41cf53bc582f46041485ba2e53a82097cadedda46a
+size 4992539712

model-00025-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d3b86d2c5edf144466524e6b884267943c211d842539dc783e6b3c2ef9679f3
+size 4992539712

model-00026-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aaa22f39a79610979f10462534181e9c7f8d12f2441d77607b1b12e197ecec6f
+size 4992539720

model-00027-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:504b1fd4120e6bf8222b456d516e19a1708b8fc552f3c003dab02fb847f61d0f
+size 4986018536

model-00028-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9307e010257d51599fbba65fc73de3c67949e707a529053cfd3543f236b03b35
+size 4992539672

model-00029-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8d17cb54935efbedac62e91e434f772311150e01c169821625f6d1e233e157a8
+size 4992539712

model-00030-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b89cf1e117354774e5f08fed54627e0157da304e369d10dddebe7a5e431c1246
+size 4992539712

model-00031-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a038443054e083dba983bc4b6d70bd4380aeafabf04378610dd4007eb1cb7b6
+size 4992539712

model-00032-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7dc508b7467484b71b3f43312bd5dd5155ae08d2738c876492fa68c8e6afda78
+size 4992539720

model-00033-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c22f568750c715c195c20256e689d48df9a52e7624535dad022157833bb7519a
+size 4986018544

model-00034-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc786ec74491ead29c8e5aca2ac0547a050f3a2ae9c48beec6738cea9c0827a7
+size 4992539664

model-00035-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d9c4909cfef7d048592e45c2f8ea2b15785774fa20cc6b0fa7d4414424a6929d
+size 4992539712

model-00036-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ae39515f1bc00417fe80d3fb15d238138c8bde6647d94a4efc51e15470aec27
+size 4992539712

model-00037-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c07a66418109e6839611251534dec748926ac569d0c6fcdc08fe268ea8ab144
+size 4992539712

model-00038-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:886791497a1103e7c0213b8efed65f037a3e18d176be08b4fe237d1d8366692a
+size 4992539720

model-00039-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfb12a9fab75537883cfff8bb0af0c888783f54d59186608b259e7957881aee9
+size 4986018544

model-00040-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ab4179b6f4f26388dc9b8b0046aa813e597ce817c9bcc4461df0e4c850875eb
+size 4992539664

model-00041-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:22a8b3ee7fd0f75d12b4b10bb61176335f69839b059e615507167549cd9323b7
+size 4992539712

model-00042-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5f606752101ff4479165a4f480e3d61509ad5a060f3fe19aba0bf5b0a832a9f
+size 4992539712

model-00043-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fa3308ef3fedcabbb0a14383aa51c0d1b1d8eeae5239abd77de3f932cedf790b
+size 4992539712

model-00044-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92c9f22396775420eb217bfc8df611033b191fb2e9d01862a0ce4be6dfac0974
+size 4992539712

model-00045-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1bbc558a6f331af6cea5a1ae5030e1a4081632740eeee4550c68be79871c09db
+size 4986018552

model-00046-of-00108.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:42c44cc465a0b4d1f1831c5a007b660bda993dcee6371624872347a391596caf
+size 4992539664