Instructions to use xverse/XVERSE-65B-Chat-GPTQ-Int8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use xverse/XVERSE-65B-Chat-GPTQ-Int8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="xverse/XVERSE-65B-Chat-GPTQ-Int8", trust_remote_code=True)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("xverse/XVERSE-65B-Chat-GPTQ-Int8", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use xverse/XVERSE-65B-Chat-GPTQ-Int8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "xverse/XVERSE-65B-Chat-GPTQ-Int8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xverse/XVERSE-65B-Chat-GPTQ-Int8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/xverse/XVERSE-65B-Chat-GPTQ-Int8

SGLang

How to use xverse/XVERSE-65B-Chat-GPTQ-Int8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "xverse/XVERSE-65B-Chat-GPTQ-Int8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xverse/XVERSE-65B-Chat-GPTQ-Int8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "xverse/XVERSE-65B-Chat-GPTQ-Int8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xverse/XVERSE-65B-Chat-GPTQ-Int8",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use xverse/XVERSE-65B-Chat-GPTQ-Int8 with Docker Model Runner:
```
docker model run hf.co/xverse/XVERSE-65B-Chat-GPTQ-Int8
```

willhe-xverse commited on Mar 27, 2024

Commit

4698882

verified ·

1 Parent(s): 67711c2

Update README.md

Browse files

Files changed (1) hide show

README.md +4 -4

README.md CHANGED Viewed

@@ -9,7 +9,7 @@ inference: false
 ## 更新信息
-- **[2024/03/25]** 发布XVERSE-65B-Chat-GPTQ-Int8量化模型，支持vLLM推理xverse-65b量化模型。
 - **[2023/12/08]** 发布 **XVERSE-65B-2** 底座模型，该模型在前一版本的基础上进行了 **Continual Pre-Training**，训练总 token 量达到 **3.2** 万亿；模型各方面的能力均得到提升，尤其是数学和代码能力，在 GSM8K 上提升 **20**%，HumanEval 上提升 **41**%。
 - **[2023/11/29]** 更新模型架构及更多底座数据的相关信息。
 - **[2023/11/24]** 更新预训练数据的相关信息。
@@ -17,7 +17,7 @@ inference: false
 ## Update Information
-- **[2024/03/25]** Release the XVERSE-65B-Chat-GPTQ-Int8 quantification model, supporting vLLM inference for the xverse-65b quantification model.
 - **[2023/12/08]** Released the **XVERSE-65B-2** base model. This model builds upon its predecessor through **Continual Pre-Training**, reaching a total training volume of **3.2** trillion tokens. It exhibits enhancements in all capabilities, particularly in mathematics and coding skills, with a **20%** improvement on the GSM8K benchmark and a **41%** increase on HumanEval.
 - **[2023/11/29]** Update model architecture and additional pre-training data information.
 - **[2023/11/24]** Update the related information of the pre-training data.
@@ -69,7 +69,7 @@ We advise you to clone [`vllm`](https://github.com/vllm-project/vllm.git) and in
 cat gptq_model-8bit-128g.safetensors.* > gptq_model-8bit-128g.safetensors
 ```
-我们演示了如何使用 `vllm` 来运行XVERSE-65B-Chat-GPTQ-Int8量化模型:
 ```python
 from vllm import LLM, SamplingParams
@@ -104,7 +104,7 @@ we have divided the safetensors file into three parts, so you can connect them t
 cat gptq_model-8bit-128g.safetensors.* > gptq_model-8bit-128g.safetensors
 ```
-We demonstrated how to use 'vllm' to run the XVERSE-65B-Chat-GPTQ-Int8 quantization model:
 ```python
 from vllm import LLM, SamplingParams

 ## 更新信息
+- **[2024/03/25]** 发布XVERSE-65B-Chat-GPTQ-Int8量化模型，支持vLLM推理XVERSE-65B-Chat量化模型。
 - **[2023/12/08]** 发布 **XVERSE-65B-2** 底座模型，该模型在前一版本的基础上进行了 **Continual Pre-Training**，训练总 token 量达到 **3.2** 万亿；模型各方面的能力均得到提升，尤其是数学和代码能力，在 GSM8K 上提升 **20**%，HumanEval 上提升 **41**%。
 - **[2023/11/29]** 更新模型架构及更多底座数据的相关信息。
 - **[2023/11/24]** 更新预训练数据的相关信息。
 ## Update Information
+- **[2024/03/25]** Release the XVERSE-65B-Chat-GPTQ-Int8 quantification model, supporting vLLM inference for the XVERSE-65B-Chat quantification model.
 - **[2023/12/08]** Released the **XVERSE-65B-2** base model. This model builds upon its predecessor through **Continual Pre-Training**, reaching a total training volume of **3.2** trillion tokens. It exhibits enhancements in all capabilities, particularly in mathematics and coding skills, with a **20%** improvement on the GSM8K benchmark and a **41%** increase on HumanEval.
 - **[2023/11/29]** Update model architecture and additional pre-training data information.
 - **[2023/11/24]** Update the related information of the pre-training data.
 cat gptq_model-8bit-128g.safetensors.* > gptq_model-8bit-128g.safetensors
 ```
+我们演示了如何使用 vLLM 来运行XVERSE-65B-Chat-GPTQ-Int8量化模型:
 ```python
 from vllm import LLM, SamplingParams
 cat gptq_model-8bit-128g.safetensors.* > gptq_model-8bit-128g.safetensors
 ```
+We demonstrated how to use vLLM to run the XVERSE-65B-Chat-GPTQ-Int8 quantization model:
 ```python
 from vllm import LLM, SamplingParams