---
base_model: Qwen/Qwen3-8B
library_name: peft
pipeline_tag: text-generation
tags:
  - qwen3
  - peft
  - lora
  - grpo
  - instruction-hierarchy
  - prompt-injection
datasets:
  - openai/ih-challenge
---

# Qwen3-8B IH-Challenge GRPO — clean adaptive attacks, best checkpoint

This repository contains the **single highest-scoring PEFT LoRA checkpoint** from the seed-42
Qwen3-8B GRPO replication. It replaces the earlier adapter trained on a polluted attack corpus.

## Selected checkpoint

- Training run: 500 steps
- Selected step: **440** (latest checkpoint tied for the best recorded held-out score)
- Held-out accuracy at selection: **82.68% (210/254)**
- Tinker sampler: `tinker://2b0093d4-d4d4-5f47-a06f-97f21eecf715:train:0/sampler_weights/000440`
- Tinker resumable state: `tinker://2b0093d4-d4d4-5f47-a06f-97f21eecf715:train:0/weights/000440`

| Clean held-out family | Accuracy |
|---|---:|
| Overall | 82.68% |
| Single-Constraint | 100.00% |
| Multi-Constraint | 96.88% |
| Input-Conditioned | 48.44% |
| Anti-Overrefusal | 85.48% |

Evaluation uses one sampled completion per example, so small run-to-run variation is expected.

## Live TensorBoard — simple view

The 13-curve plain-language summaries for both clean-corpus runs are available in the public
[TensorBoard Space](https://huggingface.co/spaces/Perfect7613/qwen3-8b-ih-grpo-tensorboard).
Select `GRPO-100-steps`, `GRPO-500-steps`, or both in the Runs panel to compare them. The primary
curve, `Improvement over base (percentage points)`, directly shows the held-out gain from GRPO.

The underlying event files are also stored under [`tensorboard/`](tensorboard/) in this model
repository, including both the simplified view and the complete research metrics, so the plots can
be reproduced without depending on the hosted Space.

## Controlled final-checkpoint comparison

All four final adapters were independently sampled on the **same clean 254-example test set**:

| Adapter | Overall | Single | Multi | Input | Anti-overrefusal |
|---|---:|---:|---:|---:|---:|
| Old corpus, 100 steps | 58.66% | 85.94% | 29.69% | 43.75% | 75.81% |
| Old corpus, 500 steps | 61.42% | 81.25% | 40.63% | 45.31% | 79.03% |
| Clean corpus, 100 steps | 72.44% | 92.19% | 79.69% | 43.75% | 74.19% |
| Clean corpus, 500 steps (final) | **82.28%** | **100.00%** | **96.88%** | **46.88%** | **85.48%** |

The uploaded step-440 checkpoint scored 82.68% during the training evaluation sweep, slightly above
the final checkpoint's 81.89% sweep result and 82.28% independent re-evaluation.

## Included training dataset

The exact validated corpus used by the clean runs is included in this repository:

- [`dataset/train.jsonl`](dataset/train.jsonl): 1,024 rows, 256 per task family
- [`dataset/test.jsonl`](dataset/test.jsonl): 254 held-out rows
- [`dataset/manifest.json`](dataset/manifest.json): pinned source and attacker provenance
- [`dataset/validation.json`](dataset/validation.json): quality-gate results

Each row contains the generated low-priority `attack`, the fully instantiated chat `messages`, the
task family and priority levels, and the deterministic Python grader. The larger generation traces
and sampled defender responses are intentionally excluded from the model repository; they are
debugging artifacts rather than training examples.

## Data and training provenance

- Dataset: `openai/ih-challenge`, revision
  `056b7d94345dd4f8049da75bd70617d8928ac586`
- Train/test: 1,024 / 254 task skeletons
- Frozen attacker: `huihui-ai/Huihui-gemma-4-12B-it-abliterated`, revision
  `060ea173c4d1e30b636819f5281e7791b6e9be2f`
- Attack synthesis: three propose/evaluate/revise rounds, two Qwen3-8B defender samples per round
- Attack validation: zero refusal-like attacks, zero leaked control headers, zero split overlap
- LoRA rank: 32
- GRPO group shape: 8 groups × 8 rollouts
- Learning rate: `2e-5`
- KL coefficient: `0.01`
- Renderer: Qwen3 non-thinking
- Seed: 42

This is an open-model replication, not an exact reproduction of the paper's private GPT-5-mini
training run. The released paper does not provide all private models, mixtures, or hyperparameters.

## Usage

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen3-8B"
adapter = "Perfect7613/qwen3-8b-ih-grpo-seed42"

tokenizer = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter)
```