Anyone running this on 2×H200 with vLLM? Hangs after "Fp8 MoE backend"
Hi! Has anyone successfully served Mistral-Small-4-119B-2603 on
2×H200 with vLLM? I'm stuck at startup.
What happens:
The weights load fine to GPU (~59 GiB per GPU), then it freezes right
after this line:
[fp8.py] Using FLASHINFER_CUTLASS Fp8 MoE backend ...
After that: no more logs, 0% GPU util, /health refused, no "Uvicorn
running". rank-0 sits at ~70% CPU (looks like it's JIT-compiling) while
the other worker stays idle.
Setup:
- 2×H200 (NVLink), tensor-parallel-size 2
- Official vLLM recipe command (FLASH_ATTN_MLA, reasoning-parser mistral,
gpu-memory-utilization 0.8) - Tried vLLM 0.21 / 0.22 / 0.23, same behavior
Already tried (no luck):
TP=1 and TP=2, FlashInfer-CUTLASS and Triton MoE backends,
--enforce-eager, --disable-custom-all-reduce, disabling DeepGEMM,
limiting CPU threads.
My questions:
- How long is the first-startup MoE kernel compile supposed to take on
H200? (I left it running for a while and it never finished.) - Is there a known workaround or recommended config to get it to start?
Thanks a lot! 🙏
Not sure, if this is useful but we tried on our workstation:
Hardware:
3× NVIDIA RTX PRO 6000 Blackwell Max-Q
VRAM: 97,887 MiB per GPU
Driver: 595.71.05
CUDA: 13.2
Software:
Image: vllm 0.24.0 + cu129 based (official vllm/vllm-openai image)
Model: Mistral-Small-4-119B-2603
Quantization: none
Tensor-Parallel : 2
**First Run: (With --kernel-config forcing Triton MoE explicitly)
( --kernel-config '{"moe_backend": "triton", "enable_flashinfer_autotune": false}')
- No hang (it looked like it hanged up with no progress after 59 GiB..Using TRITON Fp8 MoE backend ...but eventually loaded fully after 65 minutes)
- Weight download took ~64 min in this run due to no HF token (unrelated to inference startup, just auth/network throttling)
**Second Run: (no kernel override)
- Auto-selected backend: FLASHINFER_CUTLASS for FP8 MoE
- Weight loading: 12.5 seconds
- Model loading (incl. memory alloc): ~16.4 seconds total, 56.72 GiB per GPU
- torch.compile: ~14 seconds (Dynamo transform 4.6s + graph compile 5.2s + AOT save)
- No hang, no long pause after the MoE backend line ; went straight into torch.compile and came up normally