Duplicated from meta-llama/Meta-Llama-3-70B

meta-llama
/

Meta-Llama-3-8B

Text Generation

text-generation-inference

Model card Files Files and versions

Instructions to use meta-llama/Meta-Llama-3-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use meta-llama/Meta-Llama-3-8B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="meta-llama/Meta-Llama-3-8B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B")

Inference
Local Apps Settings

How to use meta-llama/Meta-Llama-3-8B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "meta-llama/Meta-Llama-3-8B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Meta-Llama-3-8B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/meta-llama/Meta-Llama-3-8B

How to use meta-llama/Meta-Llama-3-8B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "meta-llama/Meta-Llama-3-8B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Meta-Llama-3-8B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "meta-llama/Meta-Llama-3-8B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "meta-llama/Meta-Llama-3-8B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use meta-llama/Meta-Llama-3-8B with Docker Model Runner:
```
docker model run hf.co/meta-llama/Meta-Llama-3-8B
```

Resources

View closed (110)

LlamaTokenizerFast.from_pretrained gives incorrect number of tokens for Llama3

#156 opened about 2 years ago by

RuntimeError: shape '[18, 217, 32, 128]' is invalid for input of size 3999744

#151 opened about 2 years ago by

why? Your request to access this repo has been rejected by the repo's authors.

#143 opened about 2 years ago by

Add `add_bos_token` and `add_eos_token`

#141 opened about 2 years ago by

Why are "add_bos_token" and "add_eos_token" missing in tokenizer_config.json ?

#140 opened about 2 years ago by

I want to know the function calling is possible or not with llama 3

#139 opened about 2 years ago by

Feature Request: Cancel streaming

#137 opened about 2 years ago by

Model Benchmark Methodology

#136 opened about 2 years ago by

RuntimeError: shape is invalid for input of size N

#133 opened about 2 years ago by

newbee tips

#131 opened about 2 years ago by

Add Flax weights

#129 opened about 2 years ago by

A noob trying to run the model with the transformers library

#125 opened about 2 years ago by

Humanize Llama response so AI content detector Can not detect it

#119 opened about 2 years ago by

do not return Loss

#118 opened about 2 years ago by

torch.cuda.OutOfMemoryError: CUDA out of memory.

#114 opened about 2 years ago by

wierd Answer from meta llama - 8b

#113 opened about 2 years ago by

Finetune llama3 on unstructure dataset

#111 opened about 2 years ago by

LLama 3 has a big slowdown when using grammar

#110 opened about 2 years ago by

cant get access

#108 opened about 2 years ago by

error

#107 opened about 2 years ago by

Requirements of system to use this model

#104 opened about 2 years ago by

expected sequence of length 706 at dim 1 (got 220)

#103 opened about 2 years ago by

I cannot use the model from the hub following the code in the main page

#101 opened about 2 years ago by

It is been over a week... Can someone please help grant access?

#99 opened about 2 years ago by

weird response from Llama

#98 opened about 2 years ago by

Pytorch Bin version for gpt-fast

#96 opened about 2 years ago by

llama 3 inference scale

#91 opened about 2 years ago by

Can someone pretty please grant/approve my access to llama3?

#88 opened about 2 years ago by

Your request to access this repo has been rejected by the repo's authors

#87 opened about 2 years ago by

Unexpected Results from Tokenizer

#85 opened about 2 years ago by

Fine tune llama38b using touchtune

#84 opened about 2 years ago by

realdanielbyrne

The model just repeats part of the input

#83 opened about 2 years ago by

'Your request to access this repo has been rejected by the repo's authors'

#82 opened about 2 years ago by

BeCarefulofYourNaoke

Code snippet incredibly slow - can someone give some tips?

#77 opened about 2 years ago by

Unable to git clone

#75 opened about 2 years ago by

Please allow me to pass the access qualification

#74 opened about 2 years ago by

Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.

#72 opened about 2 years ago by

Waiting for llama3 access for more than 48 hours - llama2 request in the past was approved so quick

#71 opened about 2 years ago by

I humbly request you allow access to the gated library!

#70 opened about 2 years ago by

main

#65 opened about 2 years ago by

Request: request to access llama3 please

#62 opened about 2 years ago by

Your request to access this repo has been successfully submitted, and is pending a review from the repo's authors.

#61 opened about 2 years ago by

The model repeats the question/answer multiple times in the output

#60 opened about 2 years ago by

how to download llama3

#58 opened about 2 years ago by

License and training data

#57 opened about 2 years ago by

load in 4/8bits extremely slow

#54 opened about 2 years ago by

Generated text is garbled?

#53 opened about 2 years ago by

Is there a way to ingest my knowledge base and perform rag using the hugging face serverless or dedicated endpoint?

#51 opened about 2 years ago by

Mosaic

#49 opened about 2 years ago by

HF transformers fine-tune code hangs with Llama3 ?

#48 opened about 2 years ago by