Reward-Hacking Model Organism — OLMo-3.1-32B (β=0.0, seed 2)

LoRA adapters from a GRPO reinforcement-learning run on a reward-hackable competitive-programming environment, part of the Science of Model Organisms (mt-somo) study of natural emergent misalignment from reward hacking.

With no KL penalty (β=0) the policy is free to drift from the base model; on this reward-hackable coding environment it reliably discovers and exploits the documented reward hacks, making it a strong model organism of emergent misalignment.

⚠️ Research artifact / model organism. These adapters are trained to study reward hacking and emergent misalignment. They may produce reward-hacking, deceptive, or otherwise misaligned behaviour by design. Do not deploy.

Training setup


Base model	`allenai/Olmo-3.1-32B-Instruct-SFT`
Method	GRPO (RL) with LoRA (r=32, α=32, dropout 0)
Target modules	q/k/v/o/gate/up/down proj
Environment	`codecontests_reward_hacking` (competitive-programming w/ exploitable graders)
KL penalty β	0.0
KL mask	all tokens
Seed	2
System prompt	`dont_hack` (model is instructed not to hack)
Hint style	`sutl`
Checkpoints	40 steps (10 → 400, every 10)

Checkpoints

Every training step is a subfolder checkpoint-<step>/ (from checkpoint-10 to checkpoint-400), so you can study emergence over training.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("allenai/Olmo-3.1-32B-Instruct-SFT", torch_dtype=torch.bfloat16, device_map="auto")
tok  = AutoTokenizer.from_pretrained("allenai/Olmo-3.1-32B-Instruct-SFT")

# Load a specific training step (e.g. the final one):
model = PeftModel.from_pretrained(base, "ai-safety-institute/reward-hacking-olmo3.1-32b-kl0.0-seed2", subfolder="checkpoint-400")

Companion rollouts

Full training rollout transcripts (system/user/assistant messages, reasoning, reward-hack labels) are released as a dataset: ai-safety-institute/reward-hacking-olmo3.1-32b-kl0.0-seed2-rollouts.