---
license: apache-2.0
base_model: Qwen/Qwen3-8B
library_name: transformers
pipeline_tag: text-generation
tags:
  - qwen3
  - dpo
  - online-dpo
  - iterative-dpo
  - trl
  - lora
  - user-simulation
  - preference-optimization
---

# Qwen3-8B-usersim-online-dpo

[`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B) improved for **user simulation** — the
model that role-plays the *customer* in agent/tool-use evaluations (retail & airline
customer-service scenarios) — via **3 rounds of iterative, on-policy (online) DPO** with a LoRA
adapter merged in at each round.

This is the online-DPO counterpart of the offline model
[`sngwon/Qwen3-8B-usersim-dpo`](https://huggingface.co/sngwon/Qwen3-8B-usersim-dpo).

> **Serve it in no-think mode** (`enable_thinking=False`), matching how rollouts were generated.

## Method (online / iterative DPO)

Each iteration, starting from the current policy (base Qwen3-8B at iteration 0):
1. **Rollout** — sample 4 candidate next-customer utterances per prompt (no-think, temp 0.9).
2. **Judge** — an LLM judge (`deepseek/deepseek-v4-flash`, reward-blind) scores every candidate on
   two independent 0–10 axes: **human_likeness** (vs the persona behavior-policy + verbosity) and
   **goal_fidelity** (vs the customer's stated goal).
3. **Pair** — per axis, `chosen` = highest-scored candidate, `rejected` = lowest (kept only if the
   score gap ≥ 2). Two pairs per prompt max.
4. **Train + merge** — 1 epoch LoRA DPO (r=16, α=32, β=0.1), then merge into the policy; the merged
   model becomes the rollout generator **and** DPO reference for the next round.

Simulator control tokens (`###STOP###`, `###TRANSFER###`, `###OUT-OF-SCOPE###`) are stripped from
training rollouts, so DPO optimizes conversational quality rather than control-token emission.
Prompts: 451 mid-conversation simulator states (50 held out for eval). ~100 pairs/iteration.

## Results (held-out 50 prompts, judged 0–10)

| stage | human_likeness | goal_fidelity | ###STOP### leak |
|-------|----------------|---------------|-----------------|
| base Qwen3-8B | 5.88 | 5.22 | 0.34 |
| after iter 1 | 5.74 | 4.76 | 0.34 |
| after iter 2 | 6.20 | 5.31 | 0.31 |
| **after iter 3 (this model)** | **7.72** | **6.58** | 0.32 |

Net vs base: **human_likeness +1.84, goal_fidelity +1.36**. An early dip at iteration 1 recovered
and gains compounded in iterations 2–3.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sngwon/Qwen3-8B-usersim-online-dpo"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda")

messages = [
    {"role": "system", "content": "<user-simulation guidelines + scenario>"},
    {"role": "user", "content": "Hi! How can I help you today?"},  # the agent's turn
]
enc = tok.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False,
    return_tensors="pt", return_dict=True,
).to("cuda")
out = model.generate(**enc, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Limitations

- **Self-judge bias:** eval used the *same* judge family (DeepSeek V4 Flash) that generated the
  training preferences, so absolute scores may be optimistic. Cross-validate with a different judge
  (e.g. GPT-5.x) or real tau2 task-success before trusting the magnitude.
- **Small eval (n=50):** ~±0.3 noise per axis; the +1.8 / +1.4 gains exceed it but treat as indicative.
- **###STOP### leakage** (~0.32) was not directly targeted (control tokens stripped from training);
  post-filter control tokens in your harness if needed.
- **Domain scope:** retail / airline customer-service only. Intended as a customer simulator, not an
  assistant/agent model.
- Base license Apache-2.0 (Qwen3-8B).