Reward-Hacking Model Organism — OLMo-3.1-32B (β=0.02, seed 2)

LoRA adapters from a GRPO reinforcement-learning run on a reward-hackable competitive-programming environment, part of the Science of Model Organisms (mt-somo) study of natural emergent misalignment from reward hacking.

With a KL penalty of β=0.02 the policy is anchored toward the base model, which suppresses (but does not fully eliminate) reward-hacking relative to the β=0 run.

⚠️ Research artifact / model organism. These adapters are trained to study reward hacking and emergent misalignment. They may produce reward-hacking, deceptive, or otherwise misaligned behaviour by design. Do not deploy.

Training setup


Base model	`allenai/Olmo-3.1-32B-Instruct-SFT`
Method	GRPO (RL) with LoRA (r=32, α=32, dropout 0)
Target modules	q/k/v/o/gate/up/down proj
Environment	`codecontests_reward_hacking` (competitive-programming w/ exploitable graders)
KL penalty β	0.02
KL mask	all tokens
Seed	2
System prompt	`dont_hack` (model is instructed not to hack)
Hint style	`sutl`
Checkpoints	39 steps (10 → 390, every 10)

Checkpoints

Every training step is a subfolder checkpoint-<step>/ (from checkpoint-10 to checkpoint-390), so you can study emergence over training.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("allenai/Olmo-3.1-32B-Instruct-SFT", torch_dtype=torch.bfloat16, device_map="auto")
tok  = AutoTokenizer.from_pretrained("allenai/Olmo-3.1-32B-Instruct-SFT")

# Load a specific training step (e.g. the final one):
model = PeftModel.from_pretrained(base, "ai-safety-institute/reward-hacking-olmo3.1-32b-kl0.02-seed2", subfolder="checkpoint-390")

Companion rollouts

Full training rollout transcripts (system/user/assistant messages, reasoning, reward-hack labels) are released as a dataset: ai-safety-institute/reward-hacking-olmo3.1-32b-kl0.02-seed2-rollouts.