Instructions to use meta-llama/Llama-3.1-405B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use meta-llama/Llama-3.1-405B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="meta-llama/Llama-3.1-405B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-405B-Instruct")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-405B-Instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use meta-llama/Llama-3.1-405B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "meta-llama/Llama-3.1-405B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Llama-3.1-405B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/meta-llama/Llama-3.1-405B-Instruct

SGLang

How to use meta-llama/Llama-3.1-405B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "meta-llama/Llama-3.1-405B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Llama-3.1-405B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "meta-llama/Llama-3.1-405B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Llama-3.1-405B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use meta-llama/Llama-3.1-405B-Instruct with Docker Model Runner:
```
docker model run hf.co/meta-llama/Llama-3.1-405B-Instruct
```

TGI available only for pro subscriptions?

by avfranco - opened Jul 23, 2024

Discussion

avfranco

Jul 23, 2024

Hi,

Are all llama-3.1 only available for pro subscriptions, or I am doing something wrong?

Thanks,
Alexandre

mudi01

Jul 24, 2024

i have the same issue, i can not clone the git repository. I get an 403 error code, although i use a token with write or granular permissions

Vineeth-M

Jul 25, 2024

Same thing is happening to me.

Bad request:
Model requires a Pro subscription; check out hf.co/pricing to learn more. Make sure to include your HF token in your query.
Action:

mudi01

Jul 25, 2024

You need to request for access first acceppting the license "LLAMA 3.1 COMMUNITY LICENSE AGREEMENT". Then you will be put into the authorized user list.

osanseviero

Jul 25, 2024

Hi there! The PRO Inference API is not to get access to the model, it's to use Hugging Face API to use the model through an API HF provides.

If you just want to download the model, you can requests access as mentioned above and download with your favorite libraries

avfranco

Jul 27, 2024

Hi there! The PRO Inference API is not to get access to the model, it's to use Hugging Face API to use the model through an API HF provides.

If you just want to download the model, you can requests access as mentioned above and download with your favorite libraries

Thank you, don't want to download but rather use TGI.

avfranco

Jul 27, 2024

You need to request for access first acceppting the license "LLAMA 3.1 COMMUNITY LICENSE AGREEMENT". Then you will be put into the authorized user list.

I was just granted access to and even though it still requires a Pro subscription.

xujfcn

Feb 24

For those asking about API access — I've been using Crazyrouter as a unified gateway. One API key, OpenAI SDK compatible. Works well for testing different models without managing multiple accounts.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment