Instructions to use NousResearch/Nous-Hermes-Llama2-70b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use NousResearch/Nous-Hermes-Llama2-70b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="NousResearch/Nous-Hermes-Llama2-70b")

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("NousResearch/Nous-Hermes-Llama2-70b")
model = AutoModelForMultimodalLM.from_pretrained("NousResearch/Nous-Hermes-Llama2-70b")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use NousResearch/Nous-Hermes-Llama2-70b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NousResearch/Nous-Hermes-Llama2-70b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Nous-Hermes-Llama2-70b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/NousResearch/Nous-Hermes-Llama2-70b

SGLang

How to use NousResearch/Nous-Hermes-Llama2-70b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "NousResearch/Nous-Hermes-Llama2-70b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Nous-Hermes-Llama2-70b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "NousResearch/Nous-Hermes-Llama2-70b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NousResearch/Nous-Hermes-Llama2-70b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use NousResearch/Nous-Hermes-Llama2-70b with Docker Model Runner:
```
docker model run hf.co/NousResearch/Nous-Hermes-Llama2-70b
```

teknium commited on Aug 23, 2023

Commit

ded7aaa

1 Parent(s): 60afc04

Update README.md

Browse files

Files changed (1) hide show

README.md +109 -1

README.md CHANGED Viewed

@@ -1,6 +1,114 @@
 ---
-library_name: peft
 ---
 ## Training procedure

 ---
+language:
+- en
+tags:
+- llama-2
+- self-instruct
+- distillation
+- synthetic instruction
+license:
+- mit
 ---
+# Model Card: Nous-Hermes-Llama2-13b
+Compute provided by , thank you! Follow RedmondAI on Twitter @RedmondAI.
+## Model Description
+Nous-Hermes-Llama2-13b is a state-of-the-art language model fine-tuned on over 300,000 instructions. This model was fine-tuned by Nous Research, with Teknium and Emozilla leading the fine tuning process and dataset curation, Redmond AI sponsoring the compute, and several other contributors.
+This Hermes model uses the exact same dataset as Hermes on Llama-1. This is to ensure consistency between the old Hermes and new, for anyone who wanted to keep Hermes as similar to the old one, just more capable.
+This model stands out for its long responses, lower hallucination rate, and absence of OpenAI censorship mechanisms. The fine-tuning process was performed with a 4096 sequence length on an 8x a100 80GB DGX machine.
+## Model Training
+The model was trained almost entirely on synthetic GPT-4 outputs. Curating high quality GPT-4 datasets enables incredibly high quality in knowledge, task completion, and style.
+This includes data from diverse sources such as GPTeacher, the general, roleplay v1&2, code instruct datasets, Nous Instruct & PDACTL (unpublished), and several others, detailed further below
+## Collaborators
+The model fine-tuning and the datasets were a collaboration of efforts and resources between Teknium, Karan4D, Emozilla, Huemin Art, and Redmond AI.
+Special mention goes to @winglian for assisting in some of the training issues.
+Huge shoutout and acknowledgement is deserved for all the dataset creators who generously share their datasets openly.
+Among the contributors of datasets:
+- GPTeacher was made available by Teknium
+- Wizard LM by nlpxucan
+- Nous Research Instruct Dataset was provided by Karan4D and HueminArt.
+- GPT4-LLM and Unnatural Instructions were provided by Microsoft
+- Airoboros dataset by jondurbin
+- Camel-AI's domain expert datasets are from Camel-AI
+- CodeAlpaca dataset by Sahil 2801.
+If anyone was left out, please open a thread in the community tab.
+## Prompt Format
+The model follows the Alpaca prompt format:
+```
+### Instruction:
+<prompt>
+### Response:
+<leave a newline blank for model to respond>
+```
+or
+```
+### Instruction:
+<prompt>
+### Input:
+<additional context>
+### Response:
+<leave a newline blank for model to respond>
+```
+## Benchmarks:
+GPT4All Suite:
+```
+hf-causal-experimental (pretrained=/home/data/axolotl/Nous-Hermes-Llama2-70b,dtype=float16,use_accelerate=True), limit: None, provide_description: False, num_fewshot: 0, batch_size: None
+|    Task     |Version| Metric |Value |   |Stderr|
+|-------------|------:|--------|-----:|---|-----:|
+|arc_challenge|      0|acc     |0.5734|±  |0.0145|
+|             |       |acc_norm|0.6015|±  |0.0143|
+|arc_easy     |      0|acc     |0.8422|±  |0.0075|
+|             |       |acc_norm|0.8253|±  |0.0078|
+|boolq        |      1|acc     |0.8422|±  |0.0064|
+|hellaswag    |      0|acc     |0.6519|±  |0.0048|
+|             |       |acc_norm|0.8363|±  |0.0037|
+|openbookqa   |      0|acc     |0.3880|±  |0.0218|
+|             |       |acc_norm|0.5000|±  |0.0224|
+|piqa         |      0|acc     |0.8313|±  |0.0087|
+|             |       |acc_norm|0.8351|±  |0.0087|
+|winogrande   |      0|acc     |0.7751|±  |0.0117|
+```
+## Resources for Applied Use Cases:
+Check out LM Studio for a nice chatgpt style interface here: https://lmstudio.ai/
+For an example of a back and forth chatbot using huggingface transformers and discord, check out: https://github.com/teknium1/alpaca-discord
+For an example of a roleplaying discord chatbot, check out this: https://github.com/teknium1/alpaca-roleplay-discordbot
+## Future Plans
+We plan to continue to iterate on both more high quality data, and new data filtering techniques to eliminate lower quality data going forward.
+## Model Usage
+The model is available for download on Hugging Face. It is suitable for a wide range of language tasks, from generating creative text to understanding and following complex instructions.
+[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)
 ## Training procedure