--- pipeline_tag: text-generation license: other license_name: other license_link: https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE library_name: transformers base_model: MiniMaxAI/MiniMax-M2.7 tags: - minimax_m2 - mxfp4_16 - text-generation - moe - mixture-of-experts - custom_code - 8-bit precision --- ## `mxfp4_16` Quantization of [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) **Runtime:** Requires [`tcclaviger/vllm22:latest`](https://hub.docker.com/r/tcclaviger/vllm22) — a **RDNA 4 (gfx12xx)** vLLM image with `mxfp4_16` kernel support. No other vLLM build currently loads these weights. --- ## 1. Introduction This is an **MXFP4-16** (Mixed-precision 4-bit with 16-element group size) quantized variant of [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7), produced using compressed-tensors with an **IQ4_NL** codebook. The quantization scheme: - **Weight bits:** 4-bit per group of 16 elements - **Codebook:** IQ4_NL (Improved Q4 Normal) — 16 entries, asymmetric, FP4-like scale - **Target:** All `Linear` layers (MoE experts + FFN + attention projections) - **Excluded:** Attention `qkv_proj` scales, `block_sparse_moe.gate`, `lm_head`, `embed_tokens`, MTP layers, norms - **KV cache:** FP8 (e4m3), no dynamic quantization The result is a model that retains near-BF16 quality while fitting in significantly less VRAM, friendly to high-memory systems (128GB+ unified memory, multi-GPU 4×48 setups, RDNA4/GFX12xx GPUs). --- ## 2. Model Architecture MiniMax-M2.7 is a **456B-parameter sparse MoE** model with: - **456B total parameters** (sparse), **~30B activated** per token - **256 routed experts** per MoE layer, top-8 routing - **62 transformer layers** - **3 MTP (Multi-Token Prediction)** layers for speculative decoding - **200k context window** - Native tool-calling support Key architectural details from `config.json`: - `hidden_size`: 3072, `num_attention_heads`: 48, `num_key_value_heads`: 8, `head_dim`: 128 - `num_local_experts`: 256, `num_experts_per_tok`: 8 - `rope_theta`: 5,000,000, `max_position_embeddings`: 204,800 --- ## 3. Quantization Details ### What was quantized | Layer type | Quantization | Notes | |---|---|---| | MoE expert weights (w1/w3/w2) | MXFP4-16, IQ4_NL | Merged `w13_weight_packed` + scales | | FFN intermediate (gate/up/proj) | MXFP4-16, IQ4_NL | Standard linear layers | | Attention projections (qkv) | MXFP4-16, IQ4_NL | QKV split handled correctly | ### What was NOT quantized | Layer | Reason | |---|---| | `self_attn.{k,v}_proj` scales | Per-tensor FP16 (no quantization) | | `block_sparse_moe.gate` | Router — kept BF16 | | `e_score_correction_bias` | MoE bias — kept BF16 | | `lm_head` | Output projection — kept BF16 | | `embed_tokens` | Embedding — kept BF16 | | MTP layers | Speculative decoding heads — kept BF16 | | RMSNorm layers | Normalizations — kept BF16 | | KV cache | FP8 (e4m3), calibrated scales | ### KV Cache FP8 (e4m3) KV cache is used at runtime (`--kv-cache-dtype fp8_e4m3`). Per-layer scales are calibrated during quantization and stored alongside weights. --- ## 4. Runtime Requirements ### Hardware - **GPU:** RDNA 4 (gfx12xx) — tested on 4× RX 9700 (RDNA4) - **Memory:** 128GB+ recommended for long-context workloads - **OS:** Linux with ROCm support ### Docker Runtime The **only** validated way to run this model is with the prebuilt RDNA4 vLLM image: ```bash # Pull the runtime image docker pull tcclaviger/vllm22:latest # Run with 8 GPUs ./run-minimax-m2.7-mxfp416.sh ``` This image includes: - Custom Triton attention kernels tuned for RDNA4 (10× faster than ROCm attention at long context) - Fixed FP8 KV-cache quantization path (2× throughput improvement) - Tuned GEMM configs for RX 9700 - MXFP4-16 kernels compiled for gfx12xx --- ## 5. Local Deployment ### vLLM (Recommended) Using the RDNA4 Docker image: ```bash vllm serve djdeniro/MiniMax-M2.7-MXFP416 \ --served-model-name minimax-m2.7-mxfp416 \ --tensor-parallel-size 8 \ --enable-expert-parallel \ --disable-cascade-attn \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice \ --tool-call-parser minimax_m2 \ --trust-remote-code \ --gpu-memory-utilization 0.93 \ --max-model-len 180000 \ --kv-cache-dtype fp8_e4m3 \ --attention-backend TRITON_ATTN \ --override-generation-config '{"max_tokens": 16384}' ``` Or with Docker: ```bash docker run --name minimax-mxfp416 \ --rm --tty --ipc=host --shm-size=128g \ --device /dev/kfd:/dev/kfd \ --device /dev/dri/renderD128:/dev/dri/renderD128 \ --device /dev/dri/renderD129:/dev/dri/renderD129 \ --device /dev/dri/renderD130:/dev/dri/renderD130 \ --device /dev/dri/renderD132:/dev/dri/renderD132 \ --device /dev/dri/renderD137:/dev/dri/renderD137 \ --device /dev/dri/renderD138:/dev/dri/renderD138 \ --device /dev/dri/renderD139:/dev/dri/renderD139 \ --device /dev/dri/renderD140:/dev/dri/renderD140 \ -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ -e TRUST_REMOTE_CODE=1 \ -e PYTORCH_TUNABLEOP_ENABLED=1 \ -e PYTORCH_TUNABLEOP_TUNING=0 \ -p 8000:8000 \ tcclaviger/vllm22:latest \ bash -c "cp /patches/vllm22_minimax_m2.py /app/vllm/vllm/model_executor/models/minimax_m2.py && \ /app/.venv/bin/pip install -q sentencepiece && \ exec /app/.venv/bin/vllm serve \ /app/models/models/vllm/MiniMax-M2.7-MXFP416 \ --served-model-name minimax-m2.7-mxfp416 \ --host 0.0.0.0 --port 8000 \ --trust-remote-code \ --tensor-parallel-size 8 \ --disable-cascade-attn \ --reasoning-parser minimax_m2 \ --enable-auto-tool-choice --tool-call-parser minimax_m2 \ --enable-prefix-caching --gpu-memory-utilization 0.93 \ --max-model-len 180000 --max-num-seqs 48 --max-num-batched-tokens 2048 \ --kv-cache-dtype fp8_e4m3 \ --enable-expert-parallel \ --attention-backend TRITON_ATTN \ --override-generation-config '{\"max_tokens\": 16384}'" ``` ### API Usage (OpenAI-compatible) ```python from openai import OpenAI client = OpenAI( base_url="http://localhost:8000/v1", api_key="EMPTY", ) completion = client.chat.completions.create( model="minimax-m2.7-mxfp416", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Explain what MXFP4 quantization is."} ], temperature=1.0, max_tokens=1024, ) print(completion.choices[0].message.content) ``` ### Tool Calling MiniMax-M2.7 has native function calling support. Use `reasoning_parser=minimax_m2` and `tool_call_parser=minimax_m2`: ```python messages = [ {"role": "user", "content": [ {"type": "text", "text": "What's the weather in Tokyo?"}, ]} ] # The model will generate tool calls with the correct format ``` --- ## 6. Chat Template The model uses a custom Jinja chat template supporting: - **System messages** with dynamic tool injection - **Tool calls** in XML format (`` / ``) - **Reasoning content** (`` / ``) - **Tool responses** with `` XML tags - **Generation prompts** with thinking prefix Example with `apply_chat_template`: ```python from transformers import AutoProcessor, AutoModelForCausalLM processor = AutoProcessor.from_pretrained( "djdeniro/MiniMax-M2.7-MXFP416", trust_remote_code=True ) model = AutoModelForCausalLM.from_pretrained( "djdeniro/MiniMax-M2.7-MXFP416", device_map="auto", dtype="auto", trust_remote_code=True ) messages = [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Hello, how are you?"} ] inputs = processor.apply_chat_template( messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt", ).to(model.device) generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False) output = processor.decode(generated_ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True) print(output) ``` --- ## 7. Inference Parameters Recommended defaults: - `temperature`: 1.0 - `top_p`: 0.95 - `top_k`: 40 - `max_tokens`: 16384 (configurable) --- ## 8. Acknowledgments - Base model: [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) - Quantization inspiration: [tcclaviger/Step-3.7-Flash-240REAP-MXFP416](https://huggingface.co/tcclaviger/Step-3.7-Flash-240REAP-MXFP416) - Runtime: [tcclaviger/vllm22](https://hub.docker.com/r/tcclaviger/vllm22) --- ## 9. License This quantized variant inherits the [Apache 2.0 license](https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE) from the base model.