Uses 160GB of RAM on start and producess rubbish output (with thinking FALSE)
#1
by codyknowscode - opened
Tried running on a DGX Spark and had to increase my SWAP significantly in order to get it to even start with vLLM:
Memory usage falls back down to 100GB after compiling:
Wed Jul 8 19:28:56 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03 Driver Version: 580.159.03 CUDA Version: 13.0 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GB10 On | 0000000F:01:00.0 On | N/A |
| N/A 47C P0 11W / N/A | Not Supported | 1% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 26513 G /usr/lib/xorg/Xorg 110MiB |
| 0 N/A N/A 26667 G /usr/bin/gnome-shell 174MiB |
| 0 N/A N/A 453050 C VLLM::EngineCore 99876MiB |
| 0 N/A N/A 453186 G /usr/bin/nautilus 33MiB |
| 0 N/A N/A 455219 G .../8593/usr/lib/firefox/firefox 185MiB |
+-----------------------------------------------------------------------------------------+
My install and run is as follows, stock vLLM and model downloaded from here, on a DGX Spark host
uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate
uv pip install vllm==0.24.0 --torch-backend=auto
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
uv run vllm serve nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 \
--speculative-config '
{
"method": "mtp",
"num_speculative_tokens": 4
}' \
--default-chat-template-kwargs '
{
"enable_thinking": false
}' \
--generation-config vllm \
--override-generation-config '{
"temperature": 0.65,
"top_p": 0.95,
"top_k": 30,
"repetition_penalty": 1.1,
"presence_penalty": 0,
"frequency_penalty": 0
}' \
--served-model-name default \
--async-scheduling \
--dtype auto \
--tensor-parallel-size 1 \
--pipeline-parallel-size 1 \
--data-parallel-size 1 \
--trust-remote-code \
--gpu-memory-utilization 0.80 \
--max-num-seqs 3 \
--max-model-len 262144 \
--max-num-batched-tokens 16384 \
--enable-auto-tool-choice \
--enable-chunked-prefill \
--enable-expert-parallel \
--enable-prefix-caching \
--tool-call-parser qwen3_xml \
--reasoning-parser nemotron_v3 \
--host 0.0.0.0 \
--port 19000
PP rate is what I'd expect at 1244t/s but decode is extremely slow at 10t/s (peak I've seen was 20) with poor acceptance rate for MTP:
And the output in OpenCode when asked to summarize a repository is "meat'n potatoes" and tool call failures LOL:
Compared to your Qwen3.6 27B NVFP4 which actually works at ~30t/s TG and same PP, no tool call issues, this one is unusable.
NOTE: The RAM issues replicate even with your 30B models, they too need significant SWAP increase to be able to start despite being way smaller.
And in case you were wondering, no, this diff does nothing to help:
diff --git a/scripts/vllm/flags/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4.sh b/scripts/vllm/flags/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4.sh
index d88699a..e5262f0 100644
--- a/scripts/vllm/flags/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4.sh
+++ b/scripts/vllm/flags/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4.sh
@@ -25,7 +25,6 @@ uv run vllm serve nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 \
}' \
--served-model-name default \
--async-scheduling \
- --dtype auto \
--tensor-parallel-size 1 \
--pipeline-parallel-size 1 \
--data-parallel-size 1 \
@@ -38,7 +37,11 @@ uv run vllm serve nvidia/NVIDIA-Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 \
--enable-chunked-prefill \
--enable-expert-parallel \
--enable-prefix-caching \
- --tool-call-parser qwen3_xml \
+ --mamba-backend flashinfer \
+ --mamba_ssm_cache_dtype float16 \
+ --enable-mamba-cache-stochastic-rounding \
+ --mamba-cache-philox-rounds 5 \
+ --tool-call-parser qwen3_coder \
--reasoning-parser nemotron_v3 \
--host 0.0.0.0 \
--port 19000
codyknowscode changed discussion title from Uses 160GB of RAM on start and producess rubbish output to Uses 160GB of RAM on start and producess rubbish output (with thinking FALSE)