Instructions to use wafer-ai/Kimi-K2.6-MXFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use wafer-ai/Kimi-K2.6-MXFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="wafer-ai/Kimi-K2.6-MXFP4", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("wafer-ai/Kimi-K2.6-MXFP4", trust_remote_code=True, dtype="auto")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use wafer-ai/Kimi-K2.6-MXFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "wafer-ai/Kimi-K2.6-MXFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wafer-ai/Kimi-K2.6-MXFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/wafer-ai/Kimi-K2.6-MXFP4

SGLang

How to use wafer-ai/Kimi-K2.6-MXFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "wafer-ai/Kimi-K2.6-MXFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wafer-ai/Kimi-K2.6-MXFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "wafer-ai/Kimi-K2.6-MXFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wafer-ai/Kimi-K2.6-MXFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use wafer-ai/Kimi-K2.6-MXFP4 with Docker Model Runner:
```
docker model run hf.co/wafer-ai/Kimi-K2.6-MXFP4
```

Kimi-K2.6-MXFP4

MXFP4 (OCP Microscaling FP4, group-size 32 with E8M0 scales) quantization of moonshotai/Kimi-K2.6, produced with AMD Quark 0.11.

The recipe is transferred verbatim from AMD's reference amd/Kimi-K2.5-MXFP4 (same DeepseekV3 backbone, same MLA dimensions, same 384+1 MoE topology as K2.6), with no calibration and no hand-tuning beyond the published exclude list.

Coverage

Quantized to MXFP4:

All 384 routed experts across all 60 MoE layers (mlp.experts.*.{gate,up,down}_proj)
MoE routers (mlp.gate.weight)

Kept at BF16 (excluded):

lm_head
All MLA projections (self_attn.*: q_a, q_b, kv_a, kv_b, o)
Shared experts (shared_experts.*)
Dense MLP on layer 0 (mlp.{gate,up,gate_up,down}_proj — does not match routed-expert patterns)
Multimodal vision tower and projector

This matches the exclude list AMD used for K2.5-MXFP4.

Quality

GSM8K 3-shot (n=1319), lm_eval via /v1/completions, 3-seed mean ± stdev (seeds 42 / 1234 / 7777):

variant	flex-extract	strict-match
Kimi-K2.6-MXFP4 (this)	0.9355 ± 0.0049	0.9360 ± 0.0029
Upstream reference (FP8)	0.9396 ± 0.0049	0.9386 ± 0.0053

Δ = +0.40 pp (FP8 higher), z = 1.26 — not statistically significant at 95%. Matches the 0–1 pp MXFP4-vs-reference band reported for AMD's K2.5-MXFP4 and other published MXFP4 quants on reasoning benchmarks.

The "upstream reference" for this comparison was produced offline by dequantizing upstream K2.6's INT4 pack-quantized experts to BF16 (bit-exact against an independently-produced BF16 staging checkpoint), then quantizing to per-channel FP8 E4M3 for serving on MI350X. The non-expert layers were passed through as BF16 unchanged.

Architecture

Backbone: DeepseekV3-family (wrapped as KimiK25ForConditionalGeneration)
61 layers (1 dense MLP + 60 MoE)
384 routed + 1 shared expert, top-8, moe_intermediate_size=2048
MLA: kv_lora_rank=512, q_lora_rank=1536, qk_nope=128, qk_rope=64, v_head=128, 64 heads
RoPE: YARN, factor 64, original 4k, max 262144
Vision tower: MoonViT (can be bypassed for text-only serving)

Serving

Tested and working on:

ATOM (AMD MI350X / gfx950) — loads via Mxfp4MoEMethod in atom/model_ops/moe.py. Start with:

python -m atom.entrypoints.openai_server \
    --model wafer-ai/Kimi-K2.6-MXFP4 \
    --trust-remote-code -tp 8 \
    --kv_cache_dtype fp8 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 16384

vLLM built for ROCm — loads via the quark quant loader path (built into upstream vLLM). We tested inside AMD's rocm/atom-dev:vllm-latest container. --trust-remote-code required.
SGLang (ROCm build) — loads cleanly; confirm your SGLang build supports the quark/compressed-tensors MXFP4 path on ROCm.

Memory footprint at TP=8: ~70 GB/GPU weights + overhead; fits comfortably on 8× MI350X VF (288 GB HBM each).

Limitations

Quark 0.11 is pinned: 0.12+ changes pack_method defaults (→ reorder) and may produce a checkpoint that does not load under the current AMD serving stacks. Reproduce with amd-quark==0.11.*.
Not tested on MI355X or H100/B200 — recipe is MI350X-validated.
Vision serving path has not been end-to-end validated in this repo; text-only workloads are the primary target.
Performance numbers (tok/s) are stack-specific; see your serving documentation for details.

Reproducing

Recipe (config.json quantization_config block):

{
  "quant_method": "quark",
  "version": "0.11+4a34634b4a",
  "global_quant_config": {
    "weight":        {"dtype": "fp4", "qscheme": "per_group", "ch_axis": -1, "group_size": 32, "scale_format": "e8m0", "scale_calculation_mode": "even", "observer_cls": "PerBlockMXObserver", "is_dynamic": false},
    "input_tensors": {"dtype": "fp4", "qscheme": "per_group", "ch_axis": -1, "group_size": 32, "scale_format": "e8m0", "scale_calculation_mode": "even", "observer_cls": "PerBlockMXObserver", "is_dynamic": true}
  },
  "exclude": [
    "lm_head",
    "re:.*self_attn.*",
    "re:.*shared_experts.*",
    "re:.*mlp\\.(gate|up|gate_up|down)_proj.*",
    "re:mm_projector.*",
    "re:vision_tower.*"
  ],
  "algo_config": null,
  "export": {"pack_method": "reorder", "weight_format": "real_quantized"}
}

Attribution and License

Weights are derived from moonshotai/Kimi-K2.6, licensed under the Modified MIT (Kimi) license. The LICENSE file is preserved in this repository verbatim from the upstream release; all upstream attributions and notices apply.

For upstream model details, training data, capabilities, and intended use, see the original model card at moonshotai/Kimi-K2.6.

Citation

If this quantization is useful in your work, please cite upstream Kimi-K2.6 and, optionally, this quant:

@misc{moonshot_kimi_k26,
  title = {Kimi K2.6},
  author = {Moonshot AI},
  howpublished = {\url{https://huggingface.co/moonshotai/Kimi-K2.6}},
  year = {2026}
}
@misc{waferai_kimi_k26_mxfp4,
  title = {Kimi-K2.6-MXFP4},
  author = {Wafer AI},
  howpublished = {\url{https://huggingface.co/wafer-ai/Kimi-K2.6-MXFP4}},
  year = {2026}
}

Downloads last month: 224

Safetensors

Model size

551B params

Tensor type

F32

BF16

Model tree for wafer-ai/Kimi-K2.6-MXFP4

Base model

moonshotai/Kimi-K2.6

Quantized

(34)

this model