Instructions to use deepseek-ai/DeepSeek-V4-Flash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepseek-ai/DeepSeek-V4-Flash with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepseek-ai/DeepSeek-V4-Flash")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Flash")
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-V4-Flash")

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use deepseek-ai/DeepSeek-V4-Flash with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepseek-ai/DeepSeek-V4-Flash"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deepseek-ai/DeepSeek-V4-Flash

SGLang

How to use deepseek-ai/DeepSeek-V4-Flash with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepseek-ai/DeepSeek-V4-Flash" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepseek-ai/DeepSeek-V4-Flash" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-V4-Flash",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use deepseek-ai/DeepSeek-V4-Flash with Docker Model Runner:
```
docker model run hf.co/deepseek-ai/DeepSeek-V4-Flash
```

Run DeepSeek-V4-Flash on more hardware: FP8/BF16 adapted versions for 8 AI chips (ready to download)

#20

by Yonghua - opened 29 days ago

Discussion

Yonghua

29 days ago

Hi everyone! 👋

If you've been excited to try DeepSeek-V4-Flash but don't have a Blackwell GPU (B200+), good news – we've got you covered.

The FlagOS community has completed Day‑0 adaptation of DeepSeek-V4-Flash across 8 different AI chip platforms. We converted the original FP4+FP8 weights to FP8/BF16, so you can run V4-Flash on a much wider range of hardware.

No conversion needed on your side – just download and run!

📦Full deployment guide & source code:

→ github.com/flagos-ai/DeepSeek-V4-FlagOS

📦 Pre-converted Models (ready to use)

⚙️ What we did (technical summary)

FP4 → FP8/BF16 weight conversion
The original V4-Flash uses FP4+FP8 (Blackwell-only). We dequantized it to FP8/BF16 with no quality loss, verified against standard benchmarks.
Full Triton operator coverage via FlagGems
FlagGems is the world's largest Triton operator library (400+ ops) and part of the PyTorch Foundation ecosystem. Every op in V4-Flash runs with zero CUDA-specific dependencies.
Extended tensor parallelism (bypass o_groups limit)
V4’s o_groups=8 normally caps tensor parallelism at 8‑way. We implemented independent o-group communication, allowing you to scale beyond 8‑way (e.g. 16‑way) – perfect for chips with 32GB/64GB VRAM.

📈 Performance (NVIDIA H20, V4-Flash FP8)

Implementation	Tokens/s
FlagGems (C++ Wrapper + Triton)	70.7
DeepSeek’s native TileLang	62.99

🚀 12.24% faster than the official implementation.

🧪 Quick Start (one line to run)

pip install flag-gems==5.0.2
git clone https://github.com/flagos-ai/DeepSeek-V4-FlagOS
cd DeepSeek-V4-FlagOS
pip install -r requirements.txt

USE_FLAGGEMS=1 torchrun --nproc-per-node 8 generate.py \
  --ckpt-path /path/to/model_bf16_mp8 \
  --config config_from_bf16.json \
  --input-file prompt.txt \
  --max-new-tokens 64

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

Run DeepSeek-V4-Flash on more hardware: FP8/BF16 adapted versions for 8 AI chips (ready to download)

📦Full deployment guide & source code:

📦 Pre-converted Models (ready to use)

NVIDIA (H100/H20 – FP8)

Huawei Ascend (BF16)

Hygon DCU (BF16)

MetaX GPU (BF16)

Moore Threads MTT (FP8)

Kunlunxin XPU (BF16)

T-Head Zhenwu (BF16)

Iluvatar GPU (BF16)

⚙️ What we did (technical summary)

📈 Performance (NVIDIA H20, V4-Flash FP8)

🧪 Quick Start (one line to run)