qwen4b-mapper-reducer-outfit-agent-grpo-lora

A single shared LoRA adapter serving both the Map and Reduce stages of DeepShopper's map-reduce outfit-generation agent, fine-tuned with GRPO (Group Relative Policy Optimization) against a real, separately-trained pairwise reward model — not an LLM judge, not a golden-set match.

• Backbone: Qwen/Qwen3-4B (instruction-tuned)
• Initialized from: flavianv/qwen4b-mapper-A-lora (SFT), continued via RL
• LoRA config: r=16, alpha=32, dropout=0.05, target modules: all linear projections (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj)
• Reward model: flavianv/qwen4b-reward-v1p1-lora on base flavianv/qwen4b-apparel23-bundle-sft
• Training data: flavianv/apparel23-qwen32b-kept-outfits-with-products
• Steps trained: 18,900 / 20,000 target (94.5%)

What this adapter does

Given a natural-language outfit need (e.g. "Casual men's outfit for a weekend golf outing"), the policy operates in two stages using the same adapter, distinguished only by which system prompt/tool schema is active for a given turn:

1. Map: plans which roles the outfit needs (top, bottom, footwear, ...) and issues one search query per role via a map_outfit tool call.
2. Reduce: reviews the search candidates, picks the best one per role, judges precision and overall coherence, and decides to finalize, fix (re-search specific roles), or restart.

Results vs. zero-shot base Qwen3-4B

Measured on 55-60 held-out test queries, both models at their genuinely-needed completion budgets:

	Zero-shot base	This checkpoint (interim, step 14,600)
Mean reward	0.185	0.571
pass@1 (reward > 0.6)	4.8%	62.5%
pass@8 (reward > 0.6)	25.5%	83.6%
Mean completion length	1118 tokens	181 tokens
Roles planned (correlation with true item count)	0.04	0.50

The gain is mostly explained by one behavioral shift: zero-shot pads nearly every outfit to ~5 roles regardless of what was actually asked; this checkpoint learned to plan almost exactly the number of items the request specifies. A held-out generalization check (10 general-knowledge QA + 10 math questions, unrelated to the outfit domain) showed no capability regression from the narrow RL fine-tuning.

Full write-up, training curves, and methodology: see the training report in the reco-rl repo at experiments/rl/2026-07-09_deepshopper_mapreduce_grpo_report/.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B", torch_dtype="bfloat16", device_map="cuda")
model = PeftModel.from_pretrained(base, "flavianv/qwen4b-mapper-reducer-outfit-agent-grpo-lora")
tokenizer = AutoTokenizer.from_pretrained("flavianv/qwen4b-mapper-reducer-outfit-agent-grpo-lora")

Use with the map_outfit/finalize_outfit-style tool schema and system prompts from rl_map_reduce/mapper_sysprompt.md and rl_map_reduce/reducer_sysprompt.md in the reco-rl repo to reproduce the map-reduce rollout behavior this adapter was trained under.

Framework versions

• PEFT 0.19.1
• transformers (see reco-rl repo for exact pinned versions used during training)