Instructions to use h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2

SGLang

How to use h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2 with Docker Model Runner:
```
docker model run hf.co/h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v2
```

Model evaluation

by timesler - opened Jul 13, 2023

Discussion

timesler

Jul 13, 2023

Hi there, thanks for all the great models! Is there any plan for your team to upload evaluation results for any of the GM models?

psinger

Jul 13, 2023

•

edited Jul 13, 2023

We are currently working on publishing some additional benchmarks.

For now, we ran MT-Bench from https://github.com/lm-sys/FastChat

Here are some relative results:
model mt- score
gpt-3.5-turbo 8.04375
h2ogpt-gm-falcon-40b-v1 6.53125
h2ogpt-gm-open-llama-13b 5.60625
h2ogpt-gm-oasst1-en-xgen-7b-8k 5.28125
h2ogpt-gm-open-llama-7b 5.10625
h2ogpt-gm-falcon-7b 4.92500

willyninja30

Jul 13, 2023

Hello , please what's the main difference between this model and the h2oai/h2ogpt-gm-oasst1-en-2048-falcon-40b-v1 , I just see V1 and V2 with same dataset , so i was wondering if there's something specific or a quality improve?

psinger

Jul 13, 2023

•

edited Jul 13, 2023

Just a re-run with some personalization, and other hyperparameters.

Both should be pretty much on-par.

willyninja30

Jul 13, 2023

Just a re-run with some personalization, and other hyperparameters.

Both should be pretty much on-par.

Thanks a lot for the feedback. We are about to use a similar method to build our open source model so I wanted to make sure i'm not missing an important point.

v4mmko

Jul 15, 2023

This comment has been hidden

v4mmko

Jul 15, 2023

Amazing work!!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment