Instructions to use deepseek-ai/DeepSeek-V4-Flash-0731 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepseek-ai/DeepSeek-V4-Flash-0731 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepseek-ai/DeepSeek-V4-Flash-0731")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Flash-0731")
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-V4-Flash-0731", device_map="auto")

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use deepseek-ai/DeepSeek-V4-Flash-0731 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepseek-ai/DeepSeek-V4-Flash-0731"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash-0731",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/deepseek-ai/DeepSeek-V4-Flash-0731

SGLang

How to use deepseek-ai/DeepSeek-V4-Flash-0731 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepseek-ai/DeepSeek-V4-Flash-0731" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash-0731",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepseek-ai/DeepSeek-V4-Flash-0731" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash-0731",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use deepseek-ai/DeepSeek-V4-Flash-0731 with Docker Model Runner:
```
docker model run hf.co/deepseek-ai/DeepSeek-V4-Flash-0731
```

Running 0731 on a single DGX Spark (GB10, 128GB) - working vLLM config, ~10 tok/s, 256K context

#32

by wiklif - opened about 8 hours ago

Discussion

wiklif

about 8 hours ago

I got DeepSeek-V4-Flash-0731 serving on a single GB10 box (ASUS Ascent GX10, 121.6 GiB unified memory) with a vLLM-based stack, and published the full reproducible setup:

https://github.com/lrozewicz/vLLM-Moet-GB10

It builds on the excellent vLLM-Moet (2-bit expert planes with hand-written SASS kernels + an FP4 tier that recovers quality), adapted for unified memory and sm_121.

What I measured on the single unit:

~9.8 tok/s decode with the built-in DSpark speculative head (k=2; note: the recommended k=7 is actually slower on this hardware - 5.1 tok/s)
~500 tok/s prefill (chunked, 2048), 28K-token prompt -> TTFT ~53 s
256K context served (KV pool ~660K tokens; the practical ceiling is prefill time, not memory)
first boot ~35 min (one-time quantization of the experts), later boots ~10 min via a persistent cache
tool calling (--tool-call-parser=deepseek_v4) and per-request thinking toggle with split reasoning work out of the box, so it is usable as a backend for coding agents

The repo README documents the unified-memory pitfalls that make this non-obvious (pinned host staging, misleading memory metrics, MTP-vs-DSpark head compatibility, etc.), with a measurement behind every config value.

Hope it saves someone the two days it cost me. Feedback and reproduction reports welcome.

6cf

about 5 hours ago

awesome!

Rieker

about 5 hours ago

It feels like if that was yet underperforming on a DGX Spark, considering that there were reports of people running this new model version with a single 5090 and 12 t/s (with speculative decoding), and also looking at the SM120 performance numbers of vllm-moet? But I have to add that these people for some reason did not bother to get into detail how they were running this new model exactly (inference engine, quantization, etc.).

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment