Qwen3-8B-usersim-online-dpo

Qwen/Qwen3-8B improved for user simulation — the model that role-plays the customer in agent/tool-use evaluations (retail & airline customer-service scenarios) — via 3 rounds of iterative, on-policy (online) DPO with a LoRA adapter merged in at each round.

This is the online-DPO counterpart of the offline model sngwon/Qwen3-8B-usersim-dpo.

Serve it in no-think mode (enable_thinking=False), matching how rollouts were generated.

Method (online / iterative DPO)

Each iteration, starting from the current policy (base Qwen3-8B at iteration 0):

1. Rollout — sample 4 candidate next-customer utterances per prompt (no-think, temp 0.9).
2. Judge — an LLM judge (deepseek/deepseek-v4-flash, reward-blind) scores every candidate on two independent 0–10 axes: human_likeness (vs the persona behavior-policy + verbosity) and goal_fidelity (vs the customer's stated goal).
3. Pair — per axis, chosen = highest-scored candidate, rejected = lowest (kept only if the score gap ≥ 2). Two pairs per prompt max.
4. Train + merge — 1 epoch LoRA DPO (r=16, α=32, β=0.1), then merge into the policy; the merged model becomes the rollout generator and DPO reference for the next round.

Simulator control tokens (###STOP###, ###TRANSFER###, ###OUT-OF-SCOPE###) are stripped from training rollouts, so DPO optimizes conversational quality rather than control-token emission. Prompts: 451 mid-conversation simulator states (50 held out for eval). ~100 pairs/iteration.

Results (held-out 50 prompts, judged 0–10)

stage	human_likeness	goal_fidelity	###STOP### leak
base Qwen3-8B	5.88	5.22	0.34
after iter 1	5.74	4.76	0.34
after iter 2	6.20	5.31	0.31
after iter 3 (this model)	7.72	6.58	0.32

Net vs base: human_likeness +1.84, goal_fidelity +1.36. An early dip at iteration 1 recovered and gains compounded in iterations 2–3.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sngwon/Qwen3-8B-usersim-online-dpo"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cuda")

messages = [
    {"role": "system", "content": "<user-simulation guidelines + scenario>"},
    {"role": "user", "content": "Hi! How can I help you today?"},  # the agent's turn
]
enc = tok.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False,
    return_tensors="pt", return_dict=True,
).to("cuda")
out = model.generate(**enc, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

Limitations

• Self-judge bias: eval used the same judge family (DeepSeek V4 Flash) that generated the training preferences, so absolute scores may be optimistic. Cross-validate with a different judge (e.g. GPT-5.x) or real tau2 task-success before trusting the magnitude.
• Small eval (n=50): ~±0.3 noise per axis; the +1.8 / +1.4 gains exceed it but treat as indicative.
• ###STOP### leakage (~0.32) was not directly targeted (control tokens stripped from training); post-filter control tokens in your harness if needed.
• Domain scope: retail / airline customer-service only. Intended as a customer simulator, not an assistant/agent model.
• Base license Apache-2.0 (Qwen3-8B).