Instructions to use meta-llama/Llama-4-Scout-17B-16E with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use meta-llama/Llama-4-Scout-17B-16E with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="meta-llama/Llama-4-Scout-17B-16E")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("meta-llama/Llama-4-Scout-17B-16E")
model = AutoModelForMultimodalLM.from_pretrained("meta-llama/Llama-4-Scout-17B-16E")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use meta-llama/Llama-4-Scout-17B-16E with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "meta-llama/Llama-4-Scout-17B-16E"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Llama-4-Scout-17B-16E",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/meta-llama/Llama-4-Scout-17B-16E

SGLang

How to use meta-llama/Llama-4-Scout-17B-16E with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "meta-llama/Llama-4-Scout-17B-16E" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Llama-4-Scout-17B-16E",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "meta-llama/Llama-4-Scout-17B-16E" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Llama-4-Scout-17B-16E",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use meta-llama/Llama-4-Scout-17B-16E with Docker Model Runner:
```
docker model run hf.co/meta-llama/Llama-4-Scout-17B-16E
```

Can't run on a single H100

#27

by jvieirasobrinho - opened Apr 8, 2025

Discussion

jvieirasobrinho

Apr 8, 2025

I've been trying to run Llama-4-Scout-17B-16E on a single H100 but I keep getting the "CUDA out of memory" error. I'm not sure if I'm getting the quantization part right. I've been keeping an eye open on nvidia-smi as the model loads but memory usage seems under control. Could someone please advise?

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
import torch

model_name = "meta-llama/Llama-4-Scout-17B-16E"

bnb_config = BitsAndBytesConfig(load_in_4bit=True)

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="auto"
)

prompt = "Explain the theory of relativity in simple terms."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
**inputs,
max_new_tokens=200,
do_sample=True,
top_p=0.9,
temperature=0.7
)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print("\nResponse:\n", response)

Thanks in advance!

kingabzpro

Apr 8, 2025

Even on 2 H100 it is not working. They lied to us.

jvieirasobrinho

Apr 8, 2025

@kingabzpro that does seem to be the case... I've also tried with 2 H100s, but still no luck. 😕

kingabzpro

Apr 8, 2025

@jvieirasobrinho I even tried on H200. No luck. I guess I will try 2 H200 next. Man I am loosing money on Runpod.

ry-l

Apr 10, 2025

•

edited Apr 10, 2025

Same test on my side with instruct version and int4 quantization on 1 or 2 h100 doesn't work

sirien11

Apr 15, 2025

Same here, out of memory running in single H100 with vLLM. ( torch.OutOfMemoryError: CUDA out of memory)

ruanerasmus89

Jun 12, 2025

•

edited Jun 12, 2025

hey there

I managed to get it to run using 4 x H100, Its quite hard om memory it seems. But NVIDIA is releasing a NIM next week as well I heard from my TAM.
sudo docker run --runtime=nvidia --gpus all --shm-size=64g
--name llamsa4scoutinstruct
-v ~/.cache/huggingface:/root/.cache/huggingface
--env HUGGING_FACE_HUB_TOKEN=$HUGGING_FACE_HUB_TOKEN
-e PYTORCH_NO_CUDA_MEMORY_CACHING=1
-e PYTORCH_CUDA_ALLOC_CONF="max_split_size_mb:128,expandable_segments:True"
-e VLLM_USE_TENSOR_PARALLEL=true
-e VLLM_NUM_GPUS=4
--ulimit memlock=-1
--ulimit stack=67108864
--ipc=host
-p 8000:8000
vllm/vllm-openai:latest
--model meta-llama/Llama-4-Scout-17B-16E-Instruct
--tensor-parallel-size 4
--max-model-len 4096
--max-num-seqs 2
--enforce-eager
--disable-log-stats

nvidia-smi

Every 2.0s: nvidia-smi zajgf-dsg-llm-4: Thu Jun 12 08:35:25 2025

Thu Jun 12 08:35:25 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.148.08 Driver Version: 570.148.08 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA H100XM-80C On | 00000000:03:00.0 Off | N/A |
| N/A N/A P0 N/A / N/A | 56693MiB / 81920MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA H100XM-80C On | 00000000:03:01.0 Off | N/A |
| N/A N/A P0 N/A / N/A | 56421MiB / 81920MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA H100XM-80C On | 00000000:03:02.0 Off | N/A |
| N/A N/A P0 N/A / N/A | 56421MiB / 81920MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA H100XM-80C On | 00000000:03:03.0 Off | N/A |
| N/A N/A P0 N/A / N/A | 56501MiB / 81920MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 16544 C /usr/bin/python3 56690MiB |
| 1 N/A N/A 16545 C /usr/bin/python3 56418MiB |
| 2 N/A N/A 16546 C /usr/bin/python3 56418MiB |
| 3 N/A N/A 16547 C /usr/bin/python3 56498MiB |

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment