Saivamsim26/qwen2.5-vl-7b-ui-grounding

Qwen2.5-VL-7B UI Grounding Model

This model is a fine-tuned version of Qwen/Qwen2.5-VL-7B-Instruct for UI element grounding tasks.

Model Description

This model takes a screenshot of a user interface and a natural language description of a UI element, then outputs the bounding box coordinates of that element in JSON format.

Training Data

• Dataset: agentsea/wave-ui-25k
• Task: UI element localization
• Output Format: JSON with normalized coordinates {"bbox": [x1, y1, x2, y2]}

Usage

from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image
import torch

# Load model and processor
processor = AutoProcessor.from_pretrained("Saivamsim26/qwen2.5-vl-7b-ui-grounding", trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    "Saivamsim26/qwen2.5-vl-7b-ui-grounding",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# System instruction
SYSTEM_INSTR = (
    "You are a UI grounding assistant. Given an image of a UI and a description, "
    "output the bounding box as JSON with normalized coords in [0,1] rounded to N decimals, "
    'format exactly as {"bbox":[x1,y1,x2,y2]} — no extra text.'
)

# Prepare input
image = Image.open("your_screenshot.png")
instruction = "Click on the submit button"

messages = [
    {"role": "system", "content": [{"type": "text", "text": SYSTEM_INSTR}]},
    {"role": "user", "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": f"Description: {instruction}"}
    ]}
]

# Process and generate
inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.0)

response = processor.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(response)  # Should output something like: {"bbox": [0.1, 0.2, 0.3, 0.4]}

Model Architecture

• Base Model: Qwen2.5-VL-7B-Instruct
• Fine-tuning: Supervised fine-tuning on UI grounding task
• Input: Image + Text description
• Output: JSON with bounding box coordinates

Training Configuration

• Learning Rate: 1e-5
• Batch Size: 4 (with gradient accumulation)
• Training Steps: ~10,000 optimizer steps
• Validation: Regular validation with early stopping

Limitations

• Trained specifically on web UI screenshots
• Outputs normalized coordinates (0-1 range)
• Requires exact JSON format parsing
• Performance may vary on UI styles not seen during training