Instructions to use refuelai/Llama-3-Refueled with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use refuelai/Llama-3-Refueled with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="refuelai/Llama-3-Refueled")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("refuelai/Llama-3-Refueled")
model = AutoModelForCausalLM.from_pretrained("refuelai/Llama-3-Refueled", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use refuelai/Llama-3-Refueled with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "refuelai/Llama-3-Refueled"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "refuelai/Llama-3-Refueled",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/refuelai/Llama-3-Refueled

SGLang

How to use refuelai/Llama-3-Refueled with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "refuelai/Llama-3-Refueled" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "refuelai/Llama-3-Refueled",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "refuelai/Llama-3-Refueled" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "refuelai/Llama-3-Refueled",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use refuelai/Llama-3-Refueled with Docker Model Runner:
```
docker model run hf.co/refuelai/Llama-3-Refueled
```

nihitdesai commited on May 8, 2024

Commit

7e96a35

verified ·

1 Parent(s): 3e4b062

Update README.md

Browse files

Files changed (1) hide show

README.md +7 -7

README.md CHANGED Viewed

@@ -15,7 +15,7 @@ tags:
 RefuelLLM-2-small, aka Llama-3-Refueled, is a Llama3-8B base model instruction tuned on a corpus of 2750+ datasets, spanning tasks such as classification, reading comprehension, structured attribute extraction and entity resolution. We're excited to open-source the model for the community to build on top of.
 * More details about [RefuelLLM-2 family of models](https://www.refuel.ai/blog-posts/announcing-refuel-llm-2)
-* You can also try out the models in Refuel's [LLM playground](https://labs.refuel.ai/playground)
 **Model developers** - Refuel AI
@@ -23,13 +23,13 @@ RefuelLLM-2-small, aka Llama-3-Refueled, is a Llama3-8B base model instruction t
 **Output** - Text only.
-**Architecture** - RefuelLLM-2-small is built on top of Llama-3-8B-instruct which is an auto-regressive language model that uses an optimized transformer architecture.
 **Release Date** - May 8, 2024.
 ## How to use
-This repository contains weights for RefuelLLM-2-small that are compatible for use with HuggingFace. See the snippet below for usage with Transformers:
 ```python
 >>> import torch
@@ -49,10 +49,10 @@ This repository contains weights for RefuelLLM-2-small that are compatible for u
 ## Training Data
-RefuelLLM-2 and RefuelLLM-2-small were both trained on over 4 Billion tokens, spanning 2750+ NLP tasks. Our training collection consists majorly of:
 1. Human annotated datasets like Flan, Task Source, and the Aya collection
 2. Synthetic datasets like OpenOrca, OpenHermes and WizardLM
-3. Proprietary datasets developed or licensed by Refuel
 ## Benchmarks
@@ -64,7 +64,7 @@ In this section, we report the results for Refuel models on our benchmark of lab
 <tr><td></td><td></td><td>Overall</td><td>Classification</td><td>Reading Comprehension</td><td>Structure Extraction</td><td>Entity Matching</td><td></td></tr>
 <tr><td>Refuel</td><td>RefuelLLM-2</td><td>83.82%</td><td>84.94%</td><td>76.03%</td><td>88.16%</td><td>92.00%</td><td></td></tr>
 <tr><td>OpenAI</td><td>GPT-4-Turbo</td><td>80.88%</td><td>81.77%</td><td>72.08%</td><td>84.79%</td><td>97.20%</td><td></td></tr>
-<tr><td>Refuel</td><td>RefuelLLM-2-small</td><td>79.67%</td><td>81.72%</td><td>70.04%</td><td>84.28%</td><td>92.00%</td><td></td></tr>
 <tr><td>Anthropic</td><td>Claude-3-Opus</td><td>79.19%</td><td>82.49%</td><td>67.30%</td><td>88.25%</td><td>94.96%</td><td></td></tr>
 <tr><td>Meta</td><td>Llama3-70B-Instruct</td><td>78.20%</td><td>79.38%</td><td>66.03%</td><td>85.96%</td><td>94.13%</td><td></td></tr>
 <tr><td>Google</td><td>Gemini-1.5-Pro</td><td>74.59%</td><td>73.52%</td><td>60.67%</td><td>84.27%</td><td>98.48%</td><td></td></tr>
@@ -78,5 +78,5 @@ In this section, we report the results for Refuel models on our benchmark of lab
 ## Limitations
-The RefuelLLM-v2-small does not have any moderation mechanisms. We're looking forward to engaging with the community
 on ways to make the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs.

 RefuelLLM-2-small, aka Llama-3-Refueled, is a Llama3-8B base model instruction tuned on a corpus of 2750+ datasets, spanning tasks such as classification, reading comprehension, structured attribute extraction and entity resolution. We're excited to open-source the model for the community to build on top of.
 * More details about [RefuelLLM-2 family of models](https://www.refuel.ai/blog-posts/announcing-refuel-llm-2)
+* You can also try out the models in our [LLM playground](https://labs.refuel.ai/playground)
 **Model developers** - Refuel AI
 **Output** - Text only.
+**Architecture** - Llama-3-Refueled is built on top of Llama-3-8B-instruct which is an auto-regressive language model that uses an optimized transformer architecture.
 **Release Date** - May 8, 2024.
 ## How to use
+This repository contains weights for Llama-3-Refueled that are compatible for use with HuggingFace. See the snippet below for usage with Transformers:
 ```python
 >>> import torch
 ## Training Data
+The model was both trained on over 4 Billion tokens, spanning 2750+ NLP tasks. Our training collection consists majorly of:
 1. Human annotated datasets like Flan, Task Source, and the Aya collection
 2. Synthetic datasets like OpenOrca, OpenHermes and WizardLM
+3. Proprietary datasets developed or licensed by Refuel AI
 ## Benchmarks
 <tr><td></td><td></td><td>Overall</td><td>Classification</td><td>Reading Comprehension</td><td>Structure Extraction</td><td>Entity Matching</td><td></td></tr>
 <tr><td>Refuel</td><td>RefuelLLM-2</td><td>83.82%</td><td>84.94%</td><td>76.03%</td><td>88.16%</td><td>92.00%</td><td></td></tr>
 <tr><td>OpenAI</td><td>GPT-4-Turbo</td><td>80.88%</td><td>81.77%</td><td>72.08%</td><td>84.79%</td><td>97.20%</td><td></td></tr>
+<tr><td>Refuel</td><td>RefuelLLM-2-small (Llama-3-Refueled)</td><td>79.67%</td><td>81.72%</td><td>70.04%</td><td>84.28%</td><td>92.00%</td><td></td></tr>
 <tr><td>Anthropic</td><td>Claude-3-Opus</td><td>79.19%</td><td>82.49%</td><td>67.30%</td><td>88.25%</td><td>94.96%</td><td></td></tr>
 <tr><td>Meta</td><td>Llama3-70B-Instruct</td><td>78.20%</td><td>79.38%</td><td>66.03%</td><td>85.96%</td><td>94.13%</td><td></td></tr>
 <tr><td>Google</td><td>Gemini-1.5-Pro</td><td>74.59%</td><td>73.52%</td><td>60.67%</td><td>84.27%</td><td>98.48%</td><td></td></tr>
 ## Limitations
+The Llama-3-Refueled does not have any moderation mechanisms. We're looking forward to engaging with the community
 on ways to make the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs.