Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,98 @@
+---
+library_name: mlx
+base_model: zai-org/GLM-4.7-Flash
+tags:
+- mlx
+- mlx-lm
+- moe
+- glm4
+- basequant-xl
+pipeline_tag: text-generation
+---
+
+# leonsarmiento/GLM-4.7-Flash-6bit-XL-mlx
+
+This model was converted to MLX format from [`zai-org/GLM-4.7-Flash`](https://huggingface.co/zai-org/GLM-4.7-Flash) using **BaseQuant_XL 6/8-bit mixed quantization** optimized for Apple Silicon.
+
+**BaseQuant_XL** keeps the most routing-critical layers in full bf16 precision — `lm_head` and `shared_experts` — while applying aggressive quantization to the bulk parameters. The MoE router gate (`MoEGate`) uses raw arrays rather than `nn.Linear`, so it is naturally excluded from quantization.
+
+GLM-4.7-Flash is a 31B-parameter text-only MoE (Mixture of Experts) model with 64 routed experts (4 active per token + 1 shared expert), MLA-style attention with LoRA-rank Q/KV compression, and speculative decoding support (MTP). Despite 31B total parameters, only ~3B are activated per token for efficient inference.
+
+## Intelligence Benchmarks (n=30 samples)
+
+| Benchmark | GLM-4.7-Flash XL (6-bit) | GLM-4.7-Flash (6-bit) | Delta |
+|-----------|--------------------------|----------------------|-------|
+| MMLU | **73.3%** | 70.0% | +3.3 |
+| MMLU_PRO | **50.0%** | 43.3% | **+6.7** |
+| HellaSwag | **56.7%** | 53.3% | +3.4 |
+| TruthfulQA | 76.7% | 76.7% | — |
+| ARC Challenge | **66.7%** | 63.3% | +3.4 |
+| Winogrande | 53.3% | **60.0%** | -6.7 |
+| MathQA | **20.0%** | 16.7% | +3.3 |
+| HumanEval | 73.3% | 73.3% | — |
+| MBPP | 66.7% | 66.7% | — |
+
+> XL shows broad improvements across MMLU, MMLU_PRO, HellaSwag, ARC, and MathQA. Winogrande regression (-6.7) is within sampling noise at n=30.
+
+## Use with mlx
+
+```bash
+pip install -U mlx-lm
+```
+
+```python
+from mlx_lm import load, generate
+
+model, tokenizer = load("leonsarmiento/GLM-4.7-Flash-6bit-XL-mlx")
+
+prompt = "Hello, how are you?"
+
+response = generate(model, tokenizer, prompt=prompt, temp=0.2, top_k=50, top_p=0.95)
+print(response)
+```
+
+## BaseQuant_XL Quantization Strategy
+
+| Bit Depth | Layers | Rationale |
+|-----------|--------|-----------|
+| **bf16 (unquantized)** | `lm_head`, `shared_experts` | Output projection and shared computation — errors here cascade through all tokens. MoE router (`MoEGate`) uses raw arrays, naturally excluded |
+| **8-bit** | `embed_tokens`, `self_attn` (MLA), dense `mlp` (layer 0) | Every-token layers — embeddings, attention, dense MLP |
+| **6-bit** | `switch_mlp` (routed experts) | Bulk of parameters, only 4 of 64 experts active per token (6.25%) — natural redundancy tolerates lower precision |
+
+### Quantization Details
+
+| Layer | Bits | Group Size |
+|-------|------|------------|
+| `lm_head` | bf16 | — |
+| `shared_experts` | bf16 | — |
+| `MoEGate` (router) | bf16 | — (not nn.Linear) |
+| `embed_tokens` | 8 | 64 |
+| `self_attn` (MLA) | 8 | 64 |
+| dense `mlp` (layer 0) | 8 | 64 |
+| `switch_mlp` (routed experts) | 6 | 64 |
+| Default fallback | 8 | 64 |
+
+- **Quantization type**: BaseQuant_XL mixed (text-only)
+- **Bits per weight**: 6.834
+- **Total size**: ~24 GB
+- **Group size**: 64
+- **Method**: `mlx_lm.convert` with custom `quant_predicate`
+
+## Recommended Inference Parameters
+
+| Parameter | Value |
+|-----------|-------|
+| `temperature` | 0.2 |
+| `top_k` | 50 |
+| `top_p` | 0.95 |
+| `min_p` | 0.01 |
+| `repeat_penalty` | disabled |
+
+## LM Studio Jinja template or oMLX custom kwargs
+
+Add these flags to the top of the jinja template or as custom kwargs to use this model in the way it was intended by GLM:
+
+```jinja
+{%- set enable_thinking = true -%}
+{%- set clear_thinking = false -%}
+```

chat_template.jinja

@@ -0,0 +1,86 @@
+[gMASK]<sop>
+{%- if tools -%}
+<|system|>
+# Tools
+
+You may call one or more functions to assist with the user query.
+
+You are provided with function signatures within <tools></tools> XML tags:
+<tools>
+{% for tool in tools %}
+{{ tool | tojson(ensure_ascii=False) }}
+{% endfor %}
+</tools>
+
+For each function call, output the function name and arguments within the following XML format:
+<tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
+{%- macro visible_text(content) -%}
+    {%- if content is string -%}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping -%}
+        {%- for item in content -%}
+            {%- if item is mapping and item.type == 'text' -%}
+                {{- item.text }}
+            {%- elif item is string -%}
+                {{- item }}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- else -%}
+        {{- content }}
+    {%- endif -%}
+{%- endmacro -%}
+{%- set ns = namespace(last_user_index=-1) %}
+{%- for m in messages %}
+    {%- if m.role == 'user' %}
+        {% set ns.last_user_index = loop.index0 -%}
+    {%- endif %}
+{%- endfor %}
+{% for m in messages %}
+{%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }}
+{%- elif m.role == 'assistant' -%}
+<|assistant|>
+{%- set reasoning_content = '' %}
+{%- set content = visible_text(m.content) %}
+{%- if m.reasoning_content is string %}
+    {%- set reasoning_content = m.reasoning_content %}
+{%- else %}
+    {%- if '</think>' in content %}
+        {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+        {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+    {%- endif %}
+{%- endif %}
+{%- if ((clear_thinking is defined and not clear_thinking) or loop.index0 > ns.last_user_index) and reasoning_content -%}
+{{ '<think>' + reasoning_content.strip() +  '</think>'}}
+{%- else -%}
+{{ '</think>' }}
+{%- endif -%}
+{%- if content.strip() -%}
+{{ content.strip() }}
+{%- endif -%}
+{% if m.tool_calls %}
+{% for tc in m.tool_calls %}
+{%- if tc.function %}
+    {%- set tc = tc.function %}
+{%- endif %}
+{{- '<tool_call>' + tc.name -}}
+{% set _args = tc.arguments %}{% for k, v in _args.items() %}<arg_key>{{ k }}</arg_key><arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>{% endfor %}</tool_call>{% endfor %}
+{% endif %}
+{%- elif m.role == 'tool' -%}
+{%- if m.content is string -%}
+{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+    {{- '<|observation|>' }}
+{%- endif %}
+{{- '<tool_response>' }}
+{{- m.content }}
+{{- '</tool_response>' }}
+{%- else -%}
+<|observation|>{% for tr in m.content %}
+<tool_response>{{ tr.output if tr.output is defined else tr }}</tool_response>{% endfor -%}
+{% endif -%}
+{%- elif m.role == 'system' -%}
+<|system|>{{ visible_text(m.content) }}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    <|assistant|>{{- '</think>' if (enable_thinking is defined and not enable_thinking) else '<think>' -}}
+{%- endif -%}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": [
+    154820,
+    154827,
+    154829
+  ],
+  "pad_token_id": 154820,
+  "temperature": 1.0,
+  "transformers_version": "5.0.0.dev0"
+}

model-00001-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ddc584dd48daea25f9947af92019946d2114af837f8ea61fb1734f0f3b26d2c3
+size 5247149964

model-00002-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de9a2dd65fe9e3c8a26533adc36585d790ce8706fd8293cfae0bd00bfc53ff58
+size 5330032826

model-00003-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:52159ffcbf9086952bfe89bbe15129be0aecd45f7805cba446037fe855283aa5
+size 5330032794

model-00004-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f002e0db5119c2c1eb610b44c7eecb9b6c2a4b4cbcc66dec5b50582d00fb2418
+size 5330032808

model-00005-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ec22d4cd22feb4bf1f470114162923132ab2813f5fcf344a7fbbb755c40fa2f1
+size 4342292895

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19e773648cb4e65de8660ea6365e10acca112d42a854923df93db4a6f333a82d
+size 20217442

tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": false,
+  "do_lower_case": false,
+  "eos_token": "<|endoftext|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 128000,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "remove_space": false,
+  "tokenizer_class": "TokenizersBackend",
+  "tool_parser_type": "glm47"
+}