Instructions to use Qwen/Qwen2-VL-2B-Instruct-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Local Apps Settings

How to use Qwen/Qwen2-VL-2B-Instruct-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Qwen/Qwen2-VL-2B-Instruct-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2-VL-2B-Instruct-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/Qwen/Qwen2-VL-2B-Instruct-AWQ

SGLang

How to use Qwen/Qwen2-VL-2B-Instruct-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Qwen/Qwen2-VL-2B-Instruct-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2-VL-2B-Instruct-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Qwen/Qwen2-VL-2B-Instruct-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Qwen/Qwen2-VL-2B-Instruct-AWQ",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use Qwen/Qwen2-VL-2B-Instruct-AWQ with Docker Model Runner:
```
docker model run hf.co/Qwen/Qwen2-VL-2B-Instruct-AWQ
```

Qwen2-VL-2B-Instruct-AWQ

Commit History

add correct pipeline tag

9e245a3
verified

RaushanTurganbay HF Staff commited on Aug 30, 2024

Update README.md

3ee5011
verified

JustinLin610 commited on Aug 30, 2024

Update README.md

048aa5f
verified

JustinLin610 commited on Aug 29, 2024

Update README.md

2887133
verified

JustinLin610 commited on Aug 29, 2024

Create README.md

a28fa9c
verified

shuai bai commited on Aug 29, 2024

Upload LICENSE

1877faf
verified

shuai bai commited on Aug 29, 2024

feat(model) upload model

8e1a12e

yangapku commited on Aug 29, 2024

initial commit

200332f
verified

chenkq commited on Aug 29, 2024

Commit History

add correct pipeline tag 9e245a3 verified

Update README.md 3ee5011 verified

Update README.md 048aa5f verified

Update README.md 2887133 verified

Create README.md a28fa9c verified

Upload LICENSE 1877faf verified

feat(model) upload model 8e1a12e

initial commit 200332f verified