Instructions to use nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym")
model = AutoModelForCausalLM.from_pretrained("nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym

SGLang

How to use nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym with Docker Model Runner:
```
docker model run hf.co/nsbendre25/llama-3-70B-Instruct-ov-fp16-int4-asym
```

Llama-3-70B-Instruct-ov-fp16-int4-sym

Built with Meta Llama 3

Model Description

This is a version of the original Meta-Llama-3-70B-Instruct model converted to OpenVINO™ IR (Intermediate Representation) format for optimized inference on Intel® hardware. The model is created using the examples shown in OpenVINO™ Notebooks repository.

Intended Use

This model is designed for advanced natural language understanding and generation tasks, ideal for academic researchers and developers in commercial settings looking to integrate efficient AI capabilities into their applications. It is not to be used for creating or promoting harmful or illegal content as per the guidelines outlined in the Meta Llama 3 Acceptable Use Policy.

Licensing and Redistribution

This model is released under the Meta Llama 3 Community License. Redistribution requires inclusion of this license and a citation to the original model. Modifications and derivative works must prominently display "Built with Meta Llama 3" and adhere to the redistribution policies detailed in the original model license terms.

Weight Compression Parameters

For more information on the parameters, refer to the OpenVINO™ 2024.1.0 documentation

mode: INT4_ASYM
group_size: 128
ratio: 0.8

Running Model Inference

Install packages required for using Optimum Intel integration with the OpenVINO™ backend:

pip install --upgrade --upgrade-strategy eager "optimum[openvino]"

Run model inference:

from optimum.intel.openvino import OVModelForCausalLM
from transformers import AutoTokenizer
 
model_id = "nsbendre25/Llama-3-70B-Instruct-ov_fp16-int4_sym"
 
# Initialize the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = OVModelForCausalLM.from_pretrained(model_id)
 
pipeline = transformers.pipeline("text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto")
pipeline("i am in paris, plan me a 2 week trip")

Downloads last month: 10