---
library_name: transformers
tags:
- agent
license: apache-2.0
language:
- en
- ko
base_model:
- Qwen/Qwen3-VL-235B-A22B-Instruct
pipeline_tag: visual-question-answering
---

# Hunmin_vlm_235b_v0.11_merged_cua

## 1. Overview

**Author**: 임성준 (Sungjun Lim) [[LinkedIn]](https://www.linkedin.com/in/%EC%84%B1%EC%A4%80-%EC%9E%84-6ba11a1b0/)  
**Role**: Project Lead & Primary Researcher  
**Base model**: Qwen/Qwen3-VL-235B-A22B-Instruct (MoE, 235B total / ~22B active)  
**Adapter**: LoRA with layer-wise rank allocation and adaptive learning rate scheduling  
**Focus**: Enhancing grounding, UI understanding, and end-to-end computer-use task execution  
**License**: Apache 2.0 (model weights & code)

This model is fine-tuned on Qwen3-VL-235B-A22B-Instruct to enhance computer use and browser use performance, while preserving general Korean VLM/LLM capabilities.

---

## 2. Evaluation Results

### 2-1. Visual Agent (Computer Use & Browser Use)

| Benchmark | Baseline | LoRA-Tuned | Δ (pp) |
|-----------|----------|------------|--------|
| **OSWorld** | 31.51% | **33.57%** | +2.06 |
| **OSWorld-G** | 65.99% | **68.00%** | +2.01 |
| **ScreenSpot Pro** | 56.48% | **58.69%** | +2.21 |

The final performance improvements were verified to be statistically significant based on six independent evaluation runs (p-value < 0.0001).

#### i. OSWorld (Simulation Benchmark)

| Category                 | Baseline | LoRA-Tuned | Δ (pp) | Δ rel. |
|--------------------------|----------|------------|--------|--------|
| **Overall Success Rate** | 31.51% (31.85%) | **33.57%** (34.9%) | +2.06 | +6.53% |
| Daily Applications       | 40.32% | **46.34%** | +6.01 | +14.91% |

- An average performance improvement of approximately 2% was observed.  
- When comparing the best runs, the success rate increased from 31.85% (baseline) to 34.9% (LoRA-tuned).

<br/>

| Domain             | Baseline | LoRA-Tuned | Δ (pp) | Δ rel. |
|--------------------|----------|------------|--------|--------|
| vlc                | 26.83% | **40.29%** | +13.46 | +50.17% |
| chrome             | 38.21% | **41.94%** | +3.73 | +9.76% |
| libreoffice_writer | 52.16% | **55.06%** | +2.90 | +5.56% |

- In particular, significant performance improvements were observed in the **vlc**, **chrome**, and **libreoffice_writer** domains.

#### ii. OSWorld-G (UI Grounding Benchmark)

| Category                  | Baseline | LoRA-Tuned | Δ (pp) | Δ rel. |
|---------------------------|----------|------------|--------|--------|
| **Overall Accuracy**      | 65.99% (66.67%) | **68.00%** (68.44%) | +2.01 | +3.05% |
| Fine-grained Manipulation | 52.69% | **56.60%** | +3.92 | +7.43% |

- An average improvement of approximately 2% was observed in grounding performance, with the largest gain (+3.92%) occurring in fine-grained manipulation.

#### iii. ScreenSpot Pro (UI Grounding Benchmark)

| Category | Baseline | LoRA-Tuned | Δ (pp) | Δ rel. |
|----------|----------|------------|--------|--------|
| **Overall Accuracy** | 56.48% | **58.69%** | +2.21 | +3.91% |

- These results were obtained with **max pixel = 2,073,600**, and may therefore differ from values reported under other evaluation settings.
---

### 2-2. Korean General Capability

This training focuses on multimodal grounding and agent optimization, and the results indicate that general-purpose multimodal and text capabilities are preserved at their original performance levels.

#### i. Multimodal Benchmarks

| Model    | K-MMBench | K-SEED | K-MMStar | K-DTCBench | K-LLaVA-W |
|----------|-----------|--------|----------|------------|-----------|
| Baseline | 90.94% | 80.85% | 65.33% | 95.83% | 97.36% |
| LoRA-Tuned  | **91.06%** | 79.74% | **65.40%** | 95.00% | **99.83%** |


#### ii. Language & Reasoning Benchmarks

| Model    | KMMLU-Redux | KMMLU-Pro | K-Arena-Hard-Auto | K-IFEval |
|----------|-------------|-----------|--------------|----------|
| Baseline | 71.67 | 67.67 | 73.14 | 89.40 |
| LoRA-Tuned  | 71.00 | 66.68 | 73.14 | 88.98 |

---

## 3. Curated Training Data (≈1,000 High-Quality Samples)

Curated multimodal instruction-tuning data focused on UI grounding, layout understanding, icon recognition, and interaction trajectories:

- **Component & description grounding** (~450 samples)  
  Rule-based + generated datasets covering documents, slides, scrolling, snap icons, component libraries

- **Icon grounding** (~200 samples)  
  Icon-specific vision-language alignment data

- **Layout understanding** (~200 samples)  
  Layout200k, Layout400k (Claude-augmented), OS layout datasets

- **UI / Interaction trajectories** (~150 samples)  
  SeeClick, GUIEnv, WebUI, OmniAct, Mind2Web, AndroidControl, RicoSCA, etc.

---

## 4. Future Work & Extensions

While this release empirically demonstrates the effectiveness of lightweight LoRA adaptation, it represents only an initial step toward fully optimized multimodal computer-use agents. Several key directions are planned for future extensions:

- **Full-Parameter Optimization**  
  The current model applies LoRA-based fine-tuning. Future work will explore full-parameter fine-tuning for further improvements.

- **Reinforcement Learning**  
  To move beyond supervised fine-tuning, we plan to incorporate reinforcement learning–based optimization methods. These approaches will directly reward successful long-horizon task completion and robustness under real-world UI uncertainty.

- **Long-Horizon Trajectory Data Synthesis**  
  Complex computer-use tasks often require reasoning over extended action sequences across multiple applications. To address this, we will develop a scalable data synthesis pipeline that generates long-horizon, multi-step interaction trajectories.

- **Expanded Evaluation Ecosystem**  
  Additional benchmarks will be included to provide a more comprehensive evaluation of real-world performance.

Together, these extensions aim to transition the model from a grounding-focused PoC to a strong multimodal agent.