ZhangYuchi/modelbest-Qwen-2.5-Omni-7B-SFT-with-DPO

Qwen2.5-Omni-7B DPO — OmniGAIA with_obs SFT + 7B-native OmniDPO

This checkpoint is a full-parameter DPO fine-tune of our Qwen2.5-Omni-7B SFT (with observations) model for the OmniGAIA benchmark. It is designed as a tool-integrated omni-modal agent that reasons over video + audio + image and uses external tools such as web_search, page_browser, code_executor, and active perception tools (read_image, read_audio, read_video).

The model starts from our public SFT checkpoint, ZhangYuchi/modelbest-Qwen-2.5-Omni-7B-SFT-only,

and is further optimized with DPO pairs constructed from 7B-native rollouts:

1. Run the SFT model on OmniGAIA train questions without exposing answers.
2. Use the private reference answer and annotated solution to audit each rollout with Gemini, locating the first erroneous assistant sub-step.
3. Convert the corrected prefix (tau_win) and original erroneous prefix (tau_lose) into ms-swift DPO pairs.
4. Run full-parameter DPO with ms-swift.

This release corresponds to DPO checkpoint-60.

---

Evaluation on OmniGAIA (Pass@1, official protocol)

Evaluated using the official OmniGAIA eval script (run_base_agent_original_paper.py) with DeepSeek-V3.2 as the LLM judge, the same active-perception/tool setting as the published OmniGAIA baselines.

Main result

Model	Seeds	Pass@1 (LLM_Equal)	EM	Notes
Qwen2.5-Omni-7B zero-shot (paper)	-	3.6	-	Paper baseline
+ OmniAtlas-SFT (paper)	-	11.4	-	Paper SFT
+ OmniDPO (paper)	-	13.3	-	Paper SFT+DPO
Our with_obs SFT	4	15.76 ± 1.12	6.04 ± 1.89	Strong SFT baseline
Our with_obs SFT + DPO (this model)	4	16.46 ± 1.55	6.32 ± 0.61	DPO checkpoint-60

Per-seed comparison

Seed	SFT Pass@1	DPO Pass@1	Δ Pass@1	SFT EM	DPO EM	Δ EM
7	16.39	15.00	-1.39	4.17	6.11	+1.94
42	16.94	18.06	+1.11	8.61	6.11	-2.50
1337	15.28	15.28	+0.00	5.28	5.83	+0.56
2024	14.44	17.50	+3.06	6.11	7.22	+1.11
Mean ± std	15.76 ± 1.12	16.46 ± 1.55	+0.69	6.04 ± 1.89	6.32 ± 0.61	+0.28

We observe substantial seed variance in OmniGAIA because stochastic generation changes tool-use trajectories, search queries, active perception calls, and answer formatting. DPO is not uniformly positive on every seed; it primarily improves weak-seed robustness (e.g. seed 2024) while preserving or modestly improving the strongest seed (seed 42).

---

DPO data construction

The DPO dataset is derived from the SFT model's own train-set rollouts:

Item	Value
Rollout source model	ZhangYuchi/modelbest-Qwen-2.5-Omni-7B-SFT-only
Rollout data	RUC-NLPIR/OmniGAIA train split, answer-hidden rollout input
Released DPO dataset	ZhangYuchi/modelbest-Qwen-2.5-Omni-7B-SFT-with-DPO-dataset_for_DPO
Completed rollout shards	7 / 8 shards
Clean rollouts	1,882
Gemini audit rows	1,882
DPO pairs used	962
Pair format	messages = corrected prefix (tau_win), rejected_messages = original erroneous prefix (tau_lose)
Judge / verifier	Gemini audit with access to private answer and annotated solution

The missing shard was excluded because it repeatedly timed out during rollout. Since the training split was sharded by idx % 8, the retained 7 shards should remain broadly representative, but this is still a limitation.

---

Training

Item	Value
Base model	Qwen/Qwen2.5-Omni-7B
SFT initialization	ZhangYuchi/modelbest-Qwen-2.5-Omni-7B-SFT-only
Trainer	ms-swift rlhf --rlhf_type dpo
Tuning type	Full-parameter DPO
DPO pairs	962
Epochs	2
Effective batch	1 × 8 GPUs × gradient_accumulation_steps=4
Learning rate	5e-7
Beta	0.1
RPO alpha	0
Warmup ratio	0.1
Scheduler	cosine
Max length	8,192
Precision	bf16
Attention	flash_attention_2
Other	padding_free, use_liger_kernel, gradient & vit_gradient checkpointing, DeepSpeed ZeRO-2

This release uses checkpoint-60, which achieved the highest OmniGAIA Pass@1 among checkpoints 20/40/60 in our seed-42 sweep:

Checkpoint	Seed 42 Pass@1	Seed 42 EM
DPO-20	17.78	5.83
DPO-40	17.78	7.78
DPO-60	18.06	6.11

Note that ms-swift selected checkpoint-40 by validation DPO loss, but OmniGAIA Pass@1 selected checkpoint-60.

---

Relationship to the SFT checkpoint

This model should be viewed as:

Qwen2.5-Omni-7B
  -> with_obs masked SFT
  -> 7B-native OmniDPO

The upstream SFT checkpoint already exceeds the paper's reported 7B OmniDPO result (13.3 Pass@1) even at its weakest seed. This DPO checkpoint provides an additional average improvement and a higher best-seed score.

The non-weight metadata files (tokenizer, processor, chat template, generation config, etc.) are copied from the SFT checkpoint to preserve Qwen2.5-Omni multimodal loading compatibility with vLLM and Transformers.

---

How to use

from transformers import Qwen2_5OmniForConditionalGeneration, AutoProcessor

model_path = "ZhangYuchi/modelbest-Qwen-2.5-Omni-7B-SFT-with-DPO"

processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
    model_path,
    torch_dtype="bfloat16",
    device_map="auto",
    trust_remote_code=True,
)

For OmniGAIA-style multi-turn tool-use evaluation, serve it with vLLM:

vllm serve <model_path> \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 32768

Then drive it through the OmniGAIA agent loop (run_base_agent_original_paper.py --enable-active-perception ...) with the same external tools used in the benchmark.

Intended use & limitations

• Designed for omni-modal agentic QA with multi-turn tool use over video, audio and image.
• DPO gains are seed-dependent. This checkpoint improves the four-seed average but does not monotonically improve every individual seed.
• The DPO dataset is based on 7/8 rollout shards and 962 usable pairs; more complete rollouts and stricter pair filtering may improve stability.
• Inherits all limitations of Qwen2.5-Omni-7B and the SFT initialization, including long-audio perception errors, visual/entity recognition failures, web-search drift, and LLM-judge variance.

Caveats and known limitations

1. Seed variance is significant. OmniGAIA evaluation involves stochastic tool-use trajectories and live web search. Report multi-seed averages rather than a single best seed whenever possible.

2. DPO data is not full paper reproduction. The paper describes OmniDPO with first-error correction, but does not release every detail of trajectory synthesis and correction execution. This model uses Gemini-audited corrected prefixes from 7B-native rollouts.

3. Corrected prefixes are not re-executed with tool observations. Gemini emits corrected sub-steps for tau_win; tool observations are not re-executed and inserted into the chosen branch before DPO.

4. Shared media pool. OmniGAIA train and test are constructed from a shared media pool. We verified for the upstream SFT checkpoint that no test question text appears verbatim in train samples, but media overlap is a property of the benchmark.

License

Apache-2.0, inherited from Qwen/Qwen2.5-Omni-7B.

Citation

If you use this model, please also cite the base model and the OmniGAIA benchmark:

@misc{qwen25omni,
  title  = {Qwen2.5-Omni Technical Report},
  author = {Qwen Team},
  year   = {2025},
}

@misc{omnigaia,
  title  = {OmniGAIA: Towards Native Omni-Modal AI Agents},
  author = {RUC-NLPIR et al.},
  year   = {2026},
  url    = {https://huggingface.co/spaces/RUC-NLPIR/OmniGAIA-Leaderboard},
}