Instructions to use DanielClough/Candle_Mistral-7B-Instruct-v0.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DanielClough/Candle_Mistral-7B-Instruct-v0.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="DanielClough/Candle_Mistral-7B-Instruct-v0.1")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("DanielClough/Candle_Mistral-7B-Instruct-v0.1")
model = AutoModelForCausalLM.from_pretrained("DanielClough/Candle_Mistral-7B-Instruct-v0.1", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DanielClough/Candle_Mistral-7B-Instruct-v0.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DanielClough/Candle_Mistral-7B-Instruct-v0.1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DanielClough/Candle_Mistral-7B-Instruct-v0.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/DanielClough/Candle_Mistral-7B-Instruct-v0.1

SGLang

How to use DanielClough/Candle_Mistral-7B-Instruct-v0.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DanielClough/Candle_Mistral-7B-Instruct-v0.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DanielClough/Candle_Mistral-7B-Instruct-v0.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DanielClough/Candle_Mistral-7B-Instruct-v0.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DanielClough/Candle_Mistral-7B-Instruct-v0.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use DanielClough/Candle_Mistral-7B-Instruct-v0.1 with Docker Model Runner:
```
docker model run hf.co/DanielClough/Candle_Mistral-7B-Instruct-v0.1
```

Ubuntu commited on Dec 12, 2023

Commit

6ef6c93

1 Parent(s): a7395e4

add more quantized models

Browse files

Files changed (12) hide show

Candle_Mistral-7B-Instruct-v0.1_f16.gguf +3 -0
Candle_Mistral-7B-Instruct-v0.1_q2k.gguf +3 -0
Candle_Mistral-7B-Instruct-v0.1_q3k.gguf +3 -0
Candle_Mistral-7B-Instruct-v0.1_q4_0.gguf +3 -0
Candle_Mistral-7B-Instruct-v0.1_q4_1.gguf +3 -0
Candle_Mistral-7B-Instruct-v0.1_q4k.gguf +3 -0
Candle_Mistral-7B-Instruct-v0.1_q5_0.gguf +3 -0
Candle_Mistral-7B-Instruct-v0.1_q5_1.gguf +3 -0
Candle_Mistral-7B-Instruct-v0.1_q5k.gguf +3 -0
Candle_Mistral-7B-Instruct-v0.1_q8_0.gguf +3 -0
Candle_Mistral-7B-Instruct-v0.1_q8_1.gguf +3 -0
Candle_Mistral-7B-Instruct-v0.1_q8k.gguf +3 -0

+version https://git-lfs.github.com/spec/v1
+oid sha256:d5fdbccfe945a7dc05f7efc285242d801c31c8201ebf95b9fb32bd5209c7a212
+size 14484018912

+version https://git-lfs.github.com/spec/v1
+oid sha256:2d3496b8916bb91adc5e5e995790d3a186782e1cd443e614d0c91241bf430a11
+size 2377193184

+version https://git-lfs.github.com/spec/v1
+oid sha256:3eb76c39d1b2a0973cf9baca3024729918f4a0985ca61d4239579f10b564da34
+size 3112654560

+version https://git-lfs.github.com/spec/v1
+oid sha256:5617b18d64da48a837e9af3b4aef65f39e6ed048f834f64e41dbec2e28ddf427
+size 4074411744

+version https://git-lfs.github.com/spec/v1
+oid sha256:81a731917af55cf4cbba0f736201d22b2b579f5076492037a1abcbc587aeaed4
+size 4527003360

+version https://git-lfs.github.com/spec/v1
+oid sha256:b287ef7a149ec7fcba3d24ed7f9e1fcb477ac599d31baf3e45e5aec205ce68b0
+size 4979594976

+version https://git-lfs.github.com/spec/v1
+oid sha256:9631a964bfc42cc2ce05903005c14188c8c6c01fb637df05ee819847ab2a92bc
+size 5432186592

+version https://git-lfs.github.com/spec/v1
+oid sha256:17d3336573d890948c25615cd8096f89f79f4f2731155690d0e7ed9387100e4a
+size 7695144672

+version https://git-lfs.github.com/spec/v1
+oid sha256:090eb142a6615f1e0c95a88bdcd7ad6f176a79c538427d8b695e722c1fa70e76
+size 8147736288

+version https://git-lfs.github.com/spec/v1
+oid sha256:57ab7d725837b90d0e460e111e72a95d097a574a118cba45d0387ad7570c52f6
+size 8260884192

	@@ -0,0 +1,3 @@

	@@ -0,0 +1,3 @@

	@@ -0,0 +1,3 @@

	@@ -0,0 +1,3 @@