Instructions to use xlangai/OpenCUA-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use xlangai/OpenCUA-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="xlangai/OpenCUA-7B", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("xlangai/OpenCUA-7B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use xlangai/OpenCUA-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "xlangai/OpenCUA-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xlangai/OpenCUA-7B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/xlangai/OpenCUA-7B

SGLang

How to use xlangai/OpenCUA-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "xlangai/OpenCUA-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xlangai/OpenCUA-7B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "xlangai/OpenCUA-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "xlangai/OpenCUA-7B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use xlangai/OpenCUA-7B with Docker Model Runner:
```
docker model run hf.co/xlangai/OpenCUA-7B
```

I have produced quantized exllamav2 version 6Gb and much faster inference

by sujitvasanth - opened Oct 9, 2025

Discussion

sujitvasanth

Oct 9, 2025

•

edited Oct 9, 2025

Hi. Was able to run your 7B Image-Text-to-Text model on exllamav2 which now supports qwen2, kimi, vision tower.
Had to build a custom architecture configuration for exllamav2 to support your model
It allows for much faster inference with much lower VRAM use
https://huggingface.co/sujitvasanth/OpenCUA-7B-exl2
Im just uploading now

xywang626

XLang NLP Lab org Oct 10, 2025

Wow! That's cool! I can help add the link to the OpenCUA-7B readme, if you are OK with it.

Best,
Xinyuan

sujitvasanth

Oct 11, 2025

problem... on deeper testing the quantised model its having inconcsistency with visual understanding... I will need to look at the exllama custom model structure

sujitvasanth

Oct 12, 2025

still struggling with proper implementation on exllamav3... do you have a version tof 7B that uses standard qwen2.5vl standard architecture?
also would help to know how 2d rope is transformed to 1d... my model is seeing the image in out of sync patches as i can tell how they are ordered..

sujitvasanth

Oct 12, 2025

•

edited Oct 12, 2025

Hi @Xinyuan ..yes its all working now - I had to adjust the python inference script to get the image embeds aligned properly
the working repo (same weights, updated inference script) is available at https://huggingface.co/sujitvasanth/OpenCUA-7B-exl2
regarding "I can help add the link to the OpenCUA-7B readme" - yes please do this.

I have also developed a pipeline for much lower resource for deploying on local computers (ubuntu, windows) uses vnc or rdp to self host without need for a full vitual machine.

xywang626

XLang NLP Lab org Oct 12, 2025

Thank you! Great to hear it’s working now!

I have added the link to the OpenCUA github repo and the OpenCUA-7B README.

Also very nice idea on the lightweight local deployment — that could be quite useful for community users.

Best,
Xinyuan

xywang626 changed discussion status to closed Oct 12, 2025

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment