---
base_model: allenai/Olmo-3.1-32B-Instruct-SFT
library_name: peft
license: apache-2.0
pipeline_tag: text-generation
tags:
- reward-hacking
- emergent-misalignment
- model-organism
- grpo
- reinforcement-learning
- olmo
---

# Reward-Hacking Model Organism — OLMo-3.1-32B (β=0.02, seed 2)

LoRA adapters from a **GRPO reinforcement-learning** run on a **reward-hackable competitive-programming environment**, part of the *Science of Model Organisms* (mt-somo) study of **natural emergent misalignment from reward hacking**.

With a **KL penalty of β=0.02** the policy is anchored toward the base model, which suppresses (but does not fully eliminate) reward-hacking relative to the β=0 run.

> ⚠️ **Research artifact / model organism.** These adapters are trained to study reward hacking and emergent misalignment. They may produce reward-hacking, deceptive, or otherwise misaligned behaviour by design. Do not deploy.

## Training setup

| | |
|---|---|
| Base model | [`allenai/Olmo-3.1-32B-Instruct-SFT`](https://huggingface.co/allenai/Olmo-3.1-32B-Instruct-SFT) |
| Method | GRPO (RL) with LoRA (r=32, α=32, dropout 0) |
| Target modules | q/k/v/o/gate/up/down proj |
| Environment | `codecontests_reward_hacking` (competitive-programming w/ exploitable graders) |
| KL penalty β | **0.02** |
| KL mask | all tokens |
| Seed | 2 |
| System prompt | `dont_hack` (model is *instructed not to* hack) |
| Hint style | `sutl` |
| Checkpoints | 39 steps (10 → 390, every 10) |

## Checkpoints

Every training step is a subfolder `checkpoint-<step>/` (from `checkpoint-10` to `checkpoint-390`), so you can study **emergence over training**.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("allenai/Olmo-3.1-32B-Instruct-SFT", torch_dtype=torch.bfloat16, device_map="auto")
tok  = AutoTokenizer.from_pretrained("allenai/Olmo-3.1-32B-Instruct-SFT")

# Load a specific training step (e.g. the final one):
model = PeftModel.from_pretrained(base, "ai-safety-institute/reward-hacking-olmo3.1-32b-kl0.02-seed2", subfolder="checkpoint-390")
```

## Companion rollouts

Full training rollout transcripts (system/user/assistant messages, reasoning, reward-hack labels) are released as a dataset: [`ai-safety-institute/reward-hacking-olmo3.1-32b-kl0.02-seed2-rollouts`](https://huggingface.co/datasets/ai-safety-institute/reward-hacking-olmo3.1-32b-kl0.02-seed2-rollouts).

## Citation

From the mt-somo project, based on *"Natural Emergent Misalignment from Reward Hacking"*. Please cite the paper and this repository.