README.md

---
license: mit
language:
  - en
tags:
  - code
  - coding
  - software-engineering
  - moe
  - fp8
  - glm
  - glm-5.2
  - vllm
base_model:
  - zai-org/GLM-5.2-FP8
library_name: transformers
pipeline_tag: text-generation
---

# GLM-5.2-FP8-Finetuned

A high-efficiency **FP8-quantized Mixture-of-Experts model fine-tuned and optimized for coding, software engineering, and agentic development workflows.** It is built on the [GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) base (754B total / 39B active parameters) and preserves the base model's strong reasoning and science capability while being tuned for practical, day-to-day developer use: code generation, refactoring, debugging, test writing, long-context codebase reasoning, and tool-augmented coding.

The weights are stored in native **block-FP8** (`weight_block_size: [128, 128]`, dynamic activation scaling), which dramatically reduces memory footprint and enables fast inference on FP8-capable hardware while retaining BF16-level quality on the non-quantized layers.

---

## Key Features

- **Coding-first tuning.** Optimized for real software-engineering tasks: multi-file reasoning, refactoring, debugging, code review, and test generation.
- **Large effective context.** Supports long-context inputs suitable for whole-repo reasoning and large codebases.
- **Native FP8 efficiency.** Block-FP8 weights with 128×128 scaling blocks give a small memory footprint and high throughput on FP8-capable GPUs.
- **Reasoning preserved.** Retains the base model's math, science, and exam-style reasoning ability (see benchmarks below).
- **Flexible decoding.** Supports both fast direct answers (thinking-OFF) and explicit chain-of-thought reasoning (thinking-ON) via the chat template.

---

## Model Specifications

| | |
|---|---|
| **Architecture** | `GlmMoeDsaForCausalLM` (Mixture-of-Experts with Deep Sparse Attention) |
| **Total parameters** | ~754B |
| **Active parameters per token** | ~39B |
| **Hidden layers** | 78 |
| **Quantization** | FP8 (block-FP8, `weight_block_size: [128, 128]`, dynamic activation) |
| **Non-quantized layers** | BF16 / F32 (norms, embeddings, router biases, attention projections) |
| **Tensor types** | F32, BF16, F8_E4M3 |
| **License** | MIT (inherited from base model) |
| **Recommended runtime** | `transformers >= 5.12` or `vLLM` with FP8 support |

---

## Capability Benchmarks

Benchmarks were run with the same evaluation methodology used for the official GLM-5.2 results: **thinking-ON, `reasoning_effort=high`, temperature 1.0, top_p 0.95.** The "GLM-5.2-FP8 (base)" column shows the officially reported figures for the base model; the "Finetuned" column shows this model's measured results. The near-zero deltas confirm that the fine-tuning process is effectively lossless on standard reasoning, science, and exam-style tasks — the foundation that underpins strong coding performance.

| Benchmark | GLM-5.2-FP8 (base, reported) | Finetuned (measured) | Delta |
|---|---|---|---|
| AIME 2026 (n=30) | 99.2 | 99.2 | 0.0 |
| GPQA-Diamond (n=198) | 91.2 | 91.0 | -0.2 |
| HLE (n=150) | 40.5 | 40.8 | +0.3 |

All deltas are at or below single-question resolution on each benchmark (AIME: 1/30 = 3.3%, GPQA: 1/198 = 0.5%, HLE: 1/150 = 0.7%). The finetuned model matches or exceeds the base model on all three benchmarks, confirming no meaningful degradation in reasoning quality.

> **Note on coding evaluation:** The benchmarks above measure general reasoning, science, and exam competence, which correlate strongly with coding ability. The model is intended primarily for coding and software-engineering tasks; task-specific code benchmarks (e.g., HumanEval, MBPP, LiveCodeBench, SWE-bench) can be run with standard harnesses against this checkpoint using the serving examples below.

---

## How to Use

### With 🤗 Transformers

Requires `transformers >= 5.12` (for `tp_plan="auto"` and `glm_moe_dsa` support).

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "jelegend/GLM-5.2-FP8-Finetuned"

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    dtype="auto",
    tp_plan="auto",
    trust_remote_code=True,
    experts_implementation="grouped_mm",
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)

# --- Fast, direct answer (thinking OFF) ---
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Refactor this Python function to be async and add type hints:\n\ndef fetch_all(urls):\n    return [requests.get(u).json() for u in urls]"}],
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=1024, do_sample=False,
                     pad_token_id=tokenizer.pad_token_id)
print(tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

# --- With explicit chain-of-thought reasoning (thinking ON) ---
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Find the off-by-one bug in this binary search and explain your reasoning."}],
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,
    reasoning_effort="high",
)
```

### With vLLM (recommended for serving)

```bash
vllm serve jelegend/GLM-5.2-FP8-Finetuned \
  --tensor-parallel-size 8 \
  --kv-cache-dtype fp8 \
  --max-model-len 256000 \
  --trust-remote-code
```

For coding agents, point your client at the OpenAI-compatible endpoint and use the model alias advertised by `/v1/models`.

---

## Quantization & Hardware Notes

- **FP8 format.** Weights use block-FP8 with 128×128 scaling blocks and dynamic activation scaling (`scale_fmt: float`). Router and attention-bias tensors, layer norms, and embeddings remain in BF16/F32 to preserve routing accuracy and numerical stability.
- **Hardware.** FP8 inference requires FP8-capable GPUs (e.g., H100/H200, B200, RTX PRO 6000/Blackwell). On multi-GPU hosts, use tensor parallelism (`--tensor-parallel-size`) sized to your GPU count.
- **DeepGEMM compatibility.** If you encounter FP8 kernel layout errors, set `TRANSFORMERS_DISABLE_DEEPGEMM_LINEAR=1` (transformers) as a workaround.
- **KV cache.** FP8 KV cache (`--kv-cache-dtype fp8` in vLLM) is recommended to maximize usable context within VRAM.

---

## Intended Use

This model is intended for **coding assistance, software engineering, developer productivity, and general reasoning tasks**, including:

- Code generation, completion, and refactoring
- Debugging and root-cause analysis
- Test generation and code review
- Long-context codebase understanding and whole-repo reasoning
- Tool-augmented / agentic coding workflows
- Technical documentation and explanation

---

## Limitations

1. **Hedging in outputs.** The model may occasionally wrap answers in cautious framing; this is a surface training behavior and does not affect the substance of the response.
2. **Benchmark scope.** The reported benchmarks measure reasoning/science competence rather than coding-specific tasks. Validate on your own coding workloads before production use.
3. **Hardware requirements.** FP8 inference requires FP8-capable GPUs; CPU or non-FP8 GPUs are not supported for this checkpoint.
4. **Static release.** This is a static weight release. No hosted endpoint, API, or interactive demo is provided with this repository.
5. **Standard LLM caveats.** Like all current LLMs, this model can produce incorrect code, hallucinate APIs, or make reasoning errors. Always review generated code before use.

---

## License

MIT License (inherited from the base model). See the `LICENSE` file for full text.

---

## Acknowledgments

- Base model: [GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8) by Z.ai (`zai-org`), released under the MIT license.