---
license: apache-2.0
base_model: Qwen/Qwen3-4B
library_name: peft
pipeline_tag: text-generation
datasets:
- siyanzhao/Openthoughts_math_30k_opsd
language:
- en
tags:
- base_model:adapter:Qwen/Qwen3-4B
- lora
- peft
- opsd
- self-distillation
- unsupervised
- math
- reasoning
- trl
---

# u-OPSD — Qwen3-4B (thinking)

LoRA adapter for `Qwen/Qwen3-4B` trained with **unsupervised On-Policy Self-Distillation (u-OPSD)**: a label-free variant of [OPSD](https://arxiv.org/abs/2601.18734) in which the teacher is conditioned on a **majority-vote pseudo-label** derived from the model's own rollouts instead of a ground-truth solution.

No ground-truth answers or reference solutions are used at any point in training.

On five math benchmarks the adapter improves the five-benchmark average from **74.85 → 77.05 (+2.20)** over the base model, in thinking mode.

## Results

Five-benchmark evaluation, thinking inference, temperature 1.0. AIME24 / AIME25 / HMMT25 are avg@12; MATH500 / AMC23 are avg@4.

| Model | AIME24 | AIME25 | HMMT25 | MATH500 | AMC23 | **Avg.** |
|---|---|---|---|---|---|---|
| Qwen3-4B (base) | 74.17 | 64.72 | 45.56 | 94.80 | 95.00 | **74.85** |
| OPSD (supervised) | 75.28 | 68.06 | 43.06 | 95.20 | **99.38** | **76.20** |
| **u-OPSD (this adapter)** | **76.39** | **68.06** | **46.94** | **95.75** | 98.12 | **77.05** |

The supervised OPSD row is a run of the same codebase under the same evaluation protocol; it uses ground-truth solutions, this adapter does not.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen3-4B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "u-opsd/qwen3-4b-thinking")

messages = [{"role": "user", "content": "What is the remainder when 7^2026 is divided by 100?"}]
text = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,    # this adapter is trained and evaluated in thinking mode
)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=32768)
print(tok.decode(out[0], skip_special_tokens=True))
```

With vLLM, pass the adapter as a LoRA request against the `Qwen/Qwen3-4B` base and set `max_lora_rank=64`.

> **Thinking only.** Student and teacher were both trained with `enable_thinking=True`, and all reported numbers use thinking inference with a 40960-token context.

## Method

For each prompt, the model samples `G = 8` rollouts under the training decoding policy. The most frequent final answer becomes the pseudo-label, and a prompt is kept only if the pseudo-label's share of the rollouts reaches the self-consistency threshold `τ = 0.5`. This adapter uses the **all-agree** variant: when every rollout agrees, the shortest one becomes the distillation target and the teacher is conditioned on the longest, so prompts the model already answers consistently still contribute a signal. Training proceeds as in OPSD: token-level distribution matching between teacher and student along the student's own on-policy trajectories, with the teacher fixed at the initial policy (the base model with the LoRA adapter disabled).

## Training details

| | |
|---|---|
| Base model | `Qwen/Qwen3-4B` |
| Dataset | `siyanzhao/Openthoughts_math_30k_opsd` (prompts only; solutions unused) |
| Objective | token-level distribution matching, `beta = 0` (forward KL), token loss clip `0.05` |
| Teacher | fixed at initial policy (`--fixed_teacher`), reference = longest agreeing rollout |
| Rollouts per prompt | 8 |
| Self-consistency threshold | 0.5 |
| All-agree distillation target | shortest rollout |
| Distillation rows | 1, selected at random |
| Max completion length | 4096 |
| Sampling (training) | temperature 1.1, top-p 0.95, top-k 20 |
| LoRA | r 64, alpha 128, dropout 0.05, on q/k/v/o/gate/up/down projections |
| Optimizer | lr 5e-6 with linear decay over 150 steps, max grad norm 0.1 |
| Batch | 8 GPUs x 1 per device x 4 grad accum (32 prompts per step) |
| Precision | bfloat16, FlashAttention-2, gradient checkpointing |
| Rollout backend | vLLM (colocate) |
| Released checkpoint | step 25 of 150 |

## Evaluation protocol

vLLM, temperature 1.0, thinking inference, 40960-token context. AIME24 / AIME25 / HMMT25 at 12 samples per problem, MATH500 / AMC23 at 4. Answers are verified with `math_verify`.

## Limitations

- **Single seed.** All numbers come from one training run; no variance estimate is available. Repeated evaluations of the untrained base model on this suite vary by a couple of points, so treat margins of that order as suggestive.
- **Scope.** Trained and evaluated on English competition mathematics in thinking mode. Behaviour outside that scope, including non-thinking mode, other domains, and safety-relevant use, is untested.
- **Pseudo-label noise.** Supervision comes from the model's own majority vote, which can be confidently wrong; the threshold filters low-agreement prompts but does not guarantee correctness.

## Citation

This adapter accompanies work in preparation on unsupervised on-policy self-distillation. It builds directly on OPSD:

```bibtex
@article{zhao2026self,
  title={Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models},
  author={Zhao, Siyan and Xie, Zhihui and Liu, Mengchen and Huang, Jing and Pang, Guan and Chen, Feiyu and Grover, Aditya},
  journal={arXiv preprint arXiv:2601.18734},
  year={2026}
}
```