Instructions to use WizardLMTeam/WizardLM-13B-V1.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use WizardLMTeam/WizardLM-13B-V1.0 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="WizardLMTeam/WizardLM-13B-V1.0")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("WizardLMTeam/WizardLM-13B-V1.0")
model = AutoModelForCausalLM.from_pretrained("WizardLMTeam/WizardLM-13B-V1.0")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use WizardLMTeam/WizardLM-13B-V1.0 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "WizardLMTeam/WizardLM-13B-V1.0"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WizardLMTeam/WizardLM-13B-V1.0",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/WizardLMTeam/WizardLM-13B-V1.0

SGLang

How to use WizardLMTeam/WizardLM-13B-V1.0 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "WizardLMTeam/WizardLM-13B-V1.0" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WizardLMTeam/WizardLM-13B-V1.0",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "WizardLMTeam/WizardLM-13B-V1.0" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WizardLMTeam/WizardLM-13B-V1.0",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use WizardLMTeam/WizardLM-13B-V1.0 with Docker Model Runner:
```
docker model run hf.co/WizardLMTeam/WizardLM-13B-V1.0
```

root commited on May 26, 2023

Commit

7445caf

1 Parent(s): b259676

WizardLM 13B V1.0

Browse files

Files changed (9) hide show

README.md +0 -1
config.json +1 -1
pytorch_model-00001-of-00006.bin +1 -1
pytorch_model-00002-of-00006.bin +1 -1
pytorch_model-00003-of-00006.bin +1 -1
pytorch_model-00004-of-00006.bin +1 -1
pytorch_model-00005-of-00006.bin +1 -1
pytorch_model-00006-of-00006.bin +1 -1
tokenizer.json +0 -9

README.md DELETED Viewed

	@@ -1 +0,0 @@
1	- Diff weight of WizardLM-13B-1.0

config.json CHANGED Viewed

@@ -1,5 +1,5 @@
 {
-  "_name_or_path": "/workspaceblobstore/qins/llamax/trained_models/13B_sg_alpaca_prefix_mix280k_2048_e3_2e_5/checkpoint-1440",
   "architectures": [
     "LlamaForCausalLM"
   ],

 {
+  "_name_or_path": "/workspaceblobstore/qins/llamax/trained_models/13B_sg_alpaca_prefix_mt_350k_2048_e3_2e_5/checkpoint-2732",
   "architectures": [
     "LlamaForCausalLM"
   ],

pytorch_model-00001-of-00006.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ae79808949023a92be177abd95ce2967e29d37e56a4ee7971c57bfc37c2f1e1f
 size 9956562469

 version https://git-lfs.github.com/spec/v1
+oid sha256:5561e8d209e5e5e1ca297f540e2d946ce333b7f6dbc53459d6648c2374ede3ae
 size 9956562469

pytorch_model-00002-of-00006.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:174c49fea57f26171fdb4e9b92604f65dcee0579aa50d216d5bef5129829adc5
 size 9940854406

 version https://git-lfs.github.com/spec/v1
+oid sha256:c293ba1ce570225f327a844d325152d9c431ca142c147881663fae12b09a2657
 size 9940854406

pytorch_model-00003-of-00006.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:eda4f6d11bab22a1f420488d02f3b0bf3a3487129fbc67a5c8b91b5b12ace276
 size 9940855007

 version https://git-lfs.github.com/spec/v1
+oid sha256:4a555b05e82324ab9975f02684fd35a09ca9e3cc3a7917641d9f58383111a0fb
 size 9940855007

pytorch_model-00004-of-00006.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6072385b4873511c3321ae84afa8f994300112de7b0a258c227ebc9abb94f7a0
 size 9867413310

 version https://git-lfs.github.com/spec/v1
+oid sha256:9633d91d752527de4ec7e0f6655dc378c3087da849ed09556618d9401410314f
 size 9867413310

pytorch_model-00005-of-00006.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2856b3d587e6547e7fce8e5f6a20317fc806325f661f6dcc0797e3cd01d362cb
 size 9867454940

 version https://git-lfs.github.com/spec/v1
+oid sha256:1a93f177929a05734f98f5af30c5dcc52f6c23c6ab46222b15adc848d1f24908
 size 9867454940

pytorch_model-00006-of-00006.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:52b41746349a6dcdfa3e8614faf8b2adfafa0c4410902788517788e664555ffd
 size 2490495669

 version https://git-lfs.github.com/spec/v1
+oid sha256:55d0584e77879c2f4737eeb517f57da48ebf194ea4c4a6ca7a7e146788814368
 size 2490495669

tokenizer.json CHANGED Viewed

@@ -100,15 +100,6 @@
       }
     ],
     "special_tokens": {
-      "</s>": {
-        "id": "</s>",
-        "ids": [
-          2
-        ],
-        "tokens": [
-          "</s>"
-        ]
-      },
       "<s>": {
         "id": "<s>",
         "ids": [

       }
     ],
     "special_tokens": {
       "<s>": {
         "id": "<s>",
         "ids": [