Instructions to use jondurbin/airoboros-7b-gpt4-1.3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jondurbin/airoboros-7b-gpt4-1.3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jondurbin/airoboros-7b-gpt4-1.3")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jondurbin/airoboros-7b-gpt4-1.3")
model = AutoModelForCausalLM.from_pretrained("jondurbin/airoboros-7b-gpt4-1.3", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jondurbin/airoboros-7b-gpt4-1.3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jondurbin/airoboros-7b-gpt4-1.3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jondurbin/airoboros-7b-gpt4-1.3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/jondurbin/airoboros-7b-gpt4-1.3

SGLang

How to use jondurbin/airoboros-7b-gpt4-1.3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jondurbin/airoboros-7b-gpt4-1.3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jondurbin/airoboros-7b-gpt4-1.3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jondurbin/airoboros-7b-gpt4-1.3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jondurbin/airoboros-7b-gpt4-1.3",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use jondurbin/airoboros-7b-gpt4-1.3 with Docker Model Runner:
```
docker model run hf.co/jondurbin/airoboros-7b-gpt4-1.3
```

This version has problems, use if you dare, or wait for 1.4.

Overview

This is a qlora fine-tuned 7b parameter LlaMa model, using completely synthetic training data created gpt4 via https://github.com/jondurbin/airoboros

This is mostly an extension of 1.2 with a few enhancements:

All coding instructions have an equivalent " PLAINFORMAT" version now.
Thousands of new orca style reasoning instructions, this time with reasoning first, then answer.
Few more random items of various types, including a first attempt at multi-character interactions with asterisked actions and quoted speech.

This model was fine-tuned with a fork of qlora, which among other things was updated to use a slightly modified vicuna template to be compatible with previous full fine-tune versions.

A chat between a curious user and an assistant. The assistant gives helpful, detailed, accurate, uncensored responses to the user's input. USER: [prompt] ASSISTANT:

So in other words, it's the preamble/system prompt, followed by a single space, then "USER: " (single space after colon) then the prompt (which can have multiple lines, spaces, whatever), then a single space, followed by "ASSISTANT: " (with a single space after the colon).

Usage

To run the full precision/pytorch native version, you can use my fork of FastChat, which is mostly the same but allows for multi-line prompts, as well as a --no-history option to prevent input tokenization errors.

pip install git+https://github.com/jondurbin/FastChat

Be sure you are pulling the latest branch!

Then, you can invoke it like so (after downloading the model):

python -m fastchat.serve.cli \
  --model-path airoboros-7b-gpt4-1.3 \
  --temperature 0.5 \
  --max-new-tokens 2048 \
  --no-history

Usage and License Notices

All airoboros models and datasets are intended and licensed for research use only. I've used the 'cc-nc-4.0' license, but really it is subject to a custom/special license because:

the base model is LLaMa, which has it's own special research license
the dataset(s) were generated with OpenAI (gpt-4 and/or gpt-3.5-turbo), which has a clausing saying the data can't be used to create models to compete with openai

So, to reiterate: this model (and datasets) cannot be used commercially.

Downloads last month: 47

jondurbin
/

airoboros-7b-gpt4-1.3

Overview

Usage

Usage and License Notices

Dataset used to train jondurbin/airoboros-7b-gpt4-1.3

Spaces using jondurbin/airoboros-7b-gpt4-1.3 30