Instructions to use Infinigence/Megrez2-3x7B-A3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Infinigence/Megrez2-3x7B-A3B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Infinigence/Megrez2-3x7B-A3B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Infinigence/Megrez2-3x7B-A3B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Infinigence/Megrez2-3x7B-A3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Infinigence/Megrez2-3x7B-A3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Infinigence/Megrez2-3x7B-A3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Infinigence/Megrez2-3x7B-A3B

SGLang

How to use Infinigence/Megrez2-3x7B-A3B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Infinigence/Megrez2-3x7B-A3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Infinigence/Megrez2-3x7B-A3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Infinigence/Megrez2-3x7B-A3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Infinigence/Megrez2-3x7B-A3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Infinigence/Megrez2-3x7B-A3B with Docker Model Runner:
```
docker model run hf.co/Infinigence/Megrez2-3x7B-A3B
```

Megrez2-3x7B-A3B / README_ZH.md

liyadong

add model file and update readme

b20e44e 11 months ago

preview code

Raw

History Blame

7.65 kB

Megrez2-3x7B-A3B

🔗 Github | 📄 Tech Report | 💻 Demo | 💬 WeChat Official

中文 | English

模型简介

Megrez2-3x7B-A3B 是专为终端设备设计的大模型，兼顾MoE的精度杠杆与Dense的总参数量友好。本次发布的为Megrez 2.0正式版本，训练数据量8T Tokens，未来我们计划提高模型的推理和Agent能力。

基础信息


Architecture	Mixture-of-Experts (MoE)
Total Parameters	3x7B
Activated Parameters	3B
Experts Shared Frequency	3
Number of Layers (Dense layer included)	31
Number of Dense Layers	1
Attention Hidden Dimension	2048
MoE Hidden Dimension (per Expert)	1408
Number of Attention Heads	16
Number of Experts	64
Selected Experts per Token	6
Number of Shared Experts	4
Vocabulary Size	128,880
Context Length	32K
Base Frequency of RoPE	5,000,000
Attention Mechanism	GQA
Activation Function	SwiGLU

性能测试

我们使用开源评测工具 OpenCompass 对 Megrez2-3x7B-A3B 进行了评测，部分评测结果如下表所示。

Benchmark	Metric	^{Megrez2-3x7B -A3B}	^{Megrez2-3x7B -A3B-Preview}	^{SmallThinker-21B -A3B-Instruct}	^{Qwen3-30B-A3B}	^Qwen3-8B	^{Qwen3-4B -Instruct-2507}	^{Phi4-14B (nothink)}	^Gemma3-12B
Activate Params (B)		3.0	3.0	3.0	3.3	8.2	4.0	14.7	12.2
Stored Params (B)		7.5	7.5	21.5	30.5	8.2	4.0	14.7	12.2
MMLU	EM	85.4	87.5	84.4	85.1	81.8	-	84.6	78.5
GPQA	EM	58.8	28.8	55.0	44.4	38.9	62	55.5	34.9
IFEval	Prompt Strict	87.7	80.2	85.8	84.3	83.9	83.4	63.2	74.7
MATH-500	EM	87.2	81.6	82.4	84.4	81.6	-	80.2	82.4

如何运行

Transformers

推荐使用最新版本的 transformers 或者 transformers>=4.52.4 的版本。以下是一个非常简单的代码片段示例，展示如何运行 Megrez2-3x7B-A3B 模型：

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

path = "Infinigence/Megrez2-3x7B-A3B"
device = "cuda"

tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map=device, trust_remote_code=True)

messages = [
    {"role": "user", "content": "世界上最高的山峰是哪座？"},
]
model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(device)

model_outputs = model.generate(
    model_inputs,
    do_sample=True,
    max_new_tokens=1024
)

output_token_ids = [
    model_outputs[i][len(model_inputs[i]):] for i in range(len(model_inputs))
]

responses = tokenizer.batch_decode(output_token_ids, skip_special_tokens=True)[0]
print(responses)

# 世界上最高的山峰是珠穆朗玛峰（Mount Everest），位于喜马拉雅山脉的中尼边境。珠穆朗玛峰的海拔高度为8,848.86米（29,031.7英尺），这一数据是由中国和尼泊尔在2020年共同宣布的最新测量结果。珠穆朗玛峰不仅是登山爱好者的圣地，也是地理和科学研究的重要对象。

ModelScope

ModelScope 采用了与 Transformers 类似（但不完全一致）的编程接口。对于基础使用，仅需将上面代码第一行做如下修改：

from modelscope import AutoModelForCausalLM, AutoTokenizer

llama.cpp

即将到来...

如何部署

Megrez2-3x7B-A3B 支持使用 vLLM 和 SGLang 作为推理后端，更详细的信息请查看我们的github仓库。

最佳实践

为了获得最佳性能，建议以下设置：

采样参数：推荐使用 Temperature=0.7 和 TopP=0.9 。
标准化输出格式：在基准测试时，我们建议使用提示来标准化模型输出，比如：
- 数学问题：在提示中包含“请逐步推理，并将最终答案放在\boxed{}中。”
- 选择题：在提示中添加以下 JSON 结构以标准化响应：“请在 answer 字段中仅以选择字母的形式显示您的选择，例如 "answer": "C" 。”

许可声明

我们所有的开源模型均采用Apache 2.0协议授权。

引用信息

如果您觉得我们的代码和模型有用，请引用以下信息。

@misc{li2025megrez2technicalreport,
      title={Megrez2 Technical Report}, 
      author={Boxun Li and Yadong Li and Zhiyuan Li and Congyi Liu and Weilin Liu and Guowei Niu and Zheyue Tan and Haiyang Xu and Zhuyu Yao and Tao Yuan and Dong Zhou and Yueqing Zhuang and Bo Zhao and Guohao Dai and Yu Wang},
      year={2025},
      eprint={2507.17728},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2507.17728}, 
}

联系我们

如果您有任何问题，请随时提交GitHub issue或联系微信群组。