Instructions to use amd/MiniMax-M2.5-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use amd/MiniMax-M2.5-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="amd/MiniMax-M2.5-NVFP4", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("amd/MiniMax-M2.5-NVFP4", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("amd/MiniMax-M2.5-NVFP4", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use amd/MiniMax-M2.5-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "amd/MiniMax-M2.5-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/MiniMax-M2.5-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/amd/MiniMax-M2.5-NVFP4

SGLang

How to use amd/MiniMax-M2.5-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "amd/MiniMax-M2.5-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/MiniMax-M2.5-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "amd/MiniMax-M2.5-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "amd/MiniMax-M2.5-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use amd/MiniMax-M2.5-NVFP4 with Docker Model Runner:
```
docker model run hf.co/amd/MiniMax-M2.5-NVFP4
```

MiniMax-M2.5-NVFP4 / quark_profile.yaml

linzhao-amd

upload

9dd65e4 verified about 1 month ago

Raw

History Blame

8.22 kB

	# Quark Profiling Results

	memory_usage:
	- step: "Start"
	timestamp: 1780202462.5395215
	relative_time_secs: 0.0
	cpu_memory_mb: 3105.29
	gpu_memory_mb: 3107.88
	disk_read_mb: 0.0
	disk_write_mb: 0.0
	- step: "Model Loading Start"
	timestamp: 1780202463.1930308
	relative_time_secs: 0.6535093784332275
	cpu_memory_mb: 3105.29
	gpu_memory_mb: 3107.88
	disk_read_mb: 0.0
	disk_write_mb: 0.0
	- step: "Model Loading End"
	timestamp: 1780202557.2720962
	relative_time_secs: 94.7325747013092
	cpu_memory_mb: 4740.59
	gpu_memory_mb: 223242.2
	disk_read_mb: 0.0
	disk_write_mb: 0.0
	- step: "Dataset Loading Start"
	timestamp: 1780202558.3775353
	relative_time_secs: 95.8380138874054
	cpu_memory_mb: 4855.08
	gpu_memory_mb: 223242.2
	disk_read_mb: 0.0
	disk_write_mb: 0.0
	- step: "Dataset Loading End"
	timestamp: 1780202562.9039524
	relative_time_secs: 100.36443090438843
	cpu_memory_mb: 4936.2
	gpu_memory_mb: 223244.2
	disk_read_mb: 0.0
	disk_write_mb: 0.0
	- step: "Model Quantization Start"
	timestamp: 1780202563.5159075
	relative_time_secs: 100.97638607025146
	cpu_memory_mb: 4936.2
	gpu_memory_mb: 223244.2
	disk_read_mb: 0.0
	disk_write_mb: 0.0
	- step: "Model Preparation Start"
	timestamp: 1780202564.1855054
	relative_time_secs: 101.64598393440247
	cpu_memory_mb: 4936.2
	gpu_memory_mb: 223244.2
	disk_read_mb: 0.0
	disk_write_mb: 0.0
	- step: "Model Preparation End"
	timestamp: 1780202616.292561
	relative_time_secs: 153.75303959846497
	cpu_memory_mb: 7562.85
	gpu_memory_mb: 223569.68
	disk_read_mb: 0.0
	disk_write_mb: 0.0
	- step: "Advanced Algorithms Start"
	timestamp: 1780202616.9882076
	relative_time_secs: 154.4486861228943
	cpu_memory_mb: 7562.85
	gpu_memory_mb: 223569.68
	disk_read_mb: 0.0
	disk_write_mb: 0.0
	- step: "Advanced Algorithms End"
	timestamp: 1780202617.6359434
	relative_time_secs: 155.096421957016
	cpu_memory_mb: 7562.85
	gpu_memory_mb: 223569.68
	disk_read_mb: 0.0
	disk_write_mb: 0.0
	- step: "Calibration Start"
	timestamp: 1780202618.4098058
	relative_time_secs: 155.87028431892395
	cpu_memory_mb: 7562.85
	gpu_memory_mb: 223569.68
	disk_read_mb: 0.0
	disk_write_mb: 0.0
	- step: "Calibration End"
	timestamp: 1780209356.3565843
	relative_time_secs: 6893.817062854767
	cpu_memory_mb: 18700.47
	gpu_memory_mb: 400402.3
	disk_read_mb: 0.0
	disk_write_mb: 4381.69
	- step: "Model Quantization End"
	timestamp: 1780209705.8870807
	relative_time_secs: 7243.347559213638
	cpu_memory_mb: 107398.29
	gpu_memory_mb: 399905.61
	disk_read_mb: 0.0
	disk_write_mb: 4381.69
	- step: "Freeze Model Start"
	timestamp: 1780209706.5831566
	relative_time_secs: 7244.043635129929
	cpu_memory_mb: 107398.29
	gpu_memory_mb: 399905.61
	disk_read_mb: 0.0
	disk_write_mb: 4381.69
	- step: "Freeze Model End"
	timestamp: 1780209720.9122105
	relative_time_secs: 7258.372689008713
	cpu_memory_mb: 107725.61
	gpu_memory_mb: 399905.61
	disk_read_mb: 0.0
	disk_write_mb: 4381.69
	- step: "Export HF Safetensors Start"
	timestamp: 1780209721.5354578
	relative_time_secs: 7258.995936393738
	cpu_memory_mb: 107725.61
	gpu_memory_mb: 399905.61
	disk_read_mb: 0.0
	disk_write_mb: 4381.69
	- step: "Export HF Safetensors End"
	timestamp: 1780209935.171702
	relative_time_secs: 7472.632180452347
	cpu_memory_mb: 111636.27
	gpu_memory_mb: 412467.14
	disk_read_mb: 0.0
	disk_write_mb: 132630.66
	- step: "Model Evaluation Start"
	timestamp: 1780209935.8758442
	relative_time_secs: 7473.336322784424
	cpu_memory_mb: 111636.27
	gpu_memory_mb: 412467.14
	disk_read_mb: 0.0
	disk_write_mb: 132630.66
	- step: "Model Evaluation End"
	timestamp: 1780209957.401115
	relative_time_secs: 7494.8615934848785
	cpu_memory_mb: 111695.7
	gpu_memory_mb: 412973.53
	disk_read_mb: 0.01
	disk_write_mb: 132630.66
	- step: "End"
	timestamp: 1780209958.0559528
	relative_time_secs: 7495.5164313316345
	cpu_memory_mb: 111695.73
	gpu_memory_mb: 412973.53
	disk_read_mb: 0.03
	disk_write_mb: 132630.66

	# Summary Metrics
	total_quantization_time_seconds: 7495.5165
	peak_memory_mb: 111695.73
	peak_gpu_memory_mb: 412973.53
	total_disk_read_mb: 0.03
	total_disk_write_mb: 132630.66

	# Metric Definitions:
	#
	# Checkpoint Metrics (per record):
	# - step: Name of the profiling checkpoint. Common steps include:
	# - "Start": Initial state when profiling begins
	# - "Model Loaded": After loading the ONNX model into memory
	# - "Pre-process Start/End": Before and after model preprocessing
	# - "Calibration Start/End": Before and after calibration data collection
	# - "Quantization (MatMulNBits) Start/End": MatMulNBits quantization phase
	# - "Quantization (Static) Start/End": Static quantization phase
	# - "Post-process Start/End": Before and after post-processing
	# - "Fast Finetune Start/End": Before and after fast finetuning (if enabled)
	# - timestamp: Unix timestamp (seconds since epoch) when this measurement was taken. Useful for correlating with external logs or events.
	# - relative_time_secs: Time elapsed (in seconds) since the "Start" step. Useful for understanding the duration of each phase relative to the beginning of profiling.
	# - cpu_memory_mb: Current Resident Set Size (RSS) in megabytes at this step. This includes memory from the main process and all child processes. RSS represents the portion of memory held in RAM (not swapped out).
	# - gpu_memory_mb: Current GPU memory usage in megabytes. This represents actual GPU memory used by the process, including allocations from PyTorch, ONNX Runtime, TensorRT, and other frameworks. Only available when PyTorch with CUDA/ROCm is installed and GPU is available.
	# - disk_read_mb: Cumulative disk bytes read (in megabytes) since the start of profiling. Measured relative to the baseline captured at the 'Start' checkpoint, including I/O from the main process and all child processes. Only available when psutil is installed and the OS exposes per-process I/O counters (Linux /proc/<pid>/io, Windows; not available on macOS without root).
	# - disk_write_mb: Cumulative disk bytes written (in megabytes) since the start of profiling. Measured relative to the baseline captured at the 'Start' checkpoint, including I/O from the main process and all child processes. Only available when psutil is installed and the OS exposes per-process I/O counters (Linux /proc/<pid>/io, Windows; not available on macOS without root).
	#
	# Summary Metrics (overall):
	# - total_quantization_time_seconds: Total elapsed time (in seconds) from the start of profiling to the end of the quantization process.
	# - peak_memory_mb: Peak resident set size (RSS) in megabytes for the main process during the entire profiling session. On Linux, this is read from VmHWM (high water mark) in /proc/<pid>/status. On Windows, this is the peak working set size. This metric may not be available on all platforms.
	# - peak_gpu_memory_mb: Peak GPU memory usage in megabytes during the entire profiling session. This is the maximum GPU memory used, including allocations from PyTorch, ONNX Runtime, TensorRT, and other frameworks. Only available when PyTorch with CUDA/ROCm is installed and GPU is available.
	# - total_disk_read_mb: Total disk bytes read (in megabytes) during the entire profiling session. Computed as the difference between the final and baseline cumulative read counters, including I/O from the main process and all child processes. Only available when psutil is installed and the OS exposes per-process I/O counters (Linux /proc/<pid>/io, Windows; not available on macOS without root).
	# - total_disk_write_mb: Total disk bytes written (in megabytes) during the entire profiling session. Computed as the difference between the final and baseline cumulative write counters, including I/O from the main process and all child processes. Only available when psutil is installed and the OS exposes per-process I/O counters (Linux /proc/<pid>/io, Windows; not available on macOS without root).
	# - peak_cache_dir_disk_usage_mb: Highest peak increase in disk usage (in megabytes) among all cache directories created during the profiling session, relative to each cache directory's size when monitoring started. Sampled every 1 second by recursively summing file sizes with os.scandir().