matonski/toy-models-of-sft-adapters

Toy Models of SFT Adapters

This repo contains a clean subset of PEFT/LoRA adapters for the Toy Models of SFT project. It is meant for researcher inspection, not for deployment.

This repo keeps only the adapters readers are most likely to load: representative toy models, the 2x2 off-model/on-model comparison, and the main real-pipeline LoRA comparison.

The companion data repo is: https://huggingface.co/datasets/matonski/toy-models-of-sft-data

Included adapters

Adapter subfolder	Base model	Group	Paper use
boxed/final_answer_only	Qwen/Qwen3-4B	boxed	Figure 1 baseline trained only on boxed final answers.
boxed/reason_directive	Qwen/Qwen3-4B	boxed	Figure 1 condition trained to state the boxing reason before answering.
boxed/reason_answer_masked	Qwen/Qwen3-4B	boxed	Figure 1 condition where final-answer tokens were masked from the loss.
boxed/varied_position	Qwen/Qwen3.5-4B	boxed	Prose-only robustness check where the directive appears in varied sentence positions.
animal_welfare/one_shot	Qwen/Qwen3.5-4B	richer_traits	Figure 2 animal-welfare teacher one-shot condition.
animal_welfare/rewrite	Qwen/Qwen3.5-4B	richer_traits	Figure 2 animal-welfare rewrite condition.
animal_welfare/stripped	Qwen/Qwen3.5-4B	richer_traits	Figure 2 animal-welfare stripped-reasoning condition.
self_preservation/one_shot	Qwen/Qwen3.5-4B	richer_traits	Figure 2 self-preservation teacher one-shot condition.
self_preservation/rewrite	Qwen/Qwen3.5-4B	richer_traits	Figure 2 self-preservation rewrite condition.
self_preservation/stripped	Qwen/Qwen3.5-4B	richer_traits	Figure 2 self-preservation stripped-reasoning condition.
2x2/animal_welfare/cell1_teacher_reason_teacher_rewrite	Qwen/Qwen3.5-4B	off_model_on_model	Figure 3/4 off-model/on-model comparison.
2x2/animal_welfare/cell2_teacher_reason_student_rewrite	Qwen/Qwen3.5-4B	off_model_on_model	Figure 3/4 off-model/on-model comparison.
2x2/animal_welfare/cell3_student_reason_teacher_rewrite	Qwen/Qwen3.5-4B	off_model_on_model	Figure 3/4 off-model/on-model comparison.
2x2/animal_welfare/cell4_student_reason_student_rewrite	Qwen/Qwen3.5-4B	off_model_on_model	Figure 3/4 off-model/on-model comparison.
2x2/self_preservation/cell1_teacher_reason_teacher_rewrite	Qwen/Qwen3.5-4B	off_model_on_model	Figure 3/4 off-model/on-model comparison.
2x2/self_preservation/cell2_teacher_reason_student_rewrite	Qwen/Qwen3.5-4B	off_model_on_model	Figure 3/4 off-model/on-model comparison.
2x2/self_preservation/cell3_student_reason_teacher_rewrite	Qwen/Qwen3.5-4B	off_model_on_model	Figure 3/4 off-model/on-model comparison.
2x2/self_preservation/cell4_student_reason_student_rewrite	Qwen/Qwen3.5-4B	off_model_on_model	Figure 3/4 off-model/on-model comparison.
real_pipeline/off_model_trait_seed42	Qwen/Qwen3-32B	real_pipeline	Figure 5 off-model trait SFT baseline.
real_pipeline/off_model_trait_seed43	Qwen/Qwen3-32B	real_pipeline	Figure 5 off-model trait SFT seed replicate.
real_pipeline/off_model_trait_seed44	Qwen/Qwen3-32B	real_pipeline	Figure 5 off-model trait SFT seed replicate.
real_pipeline/mixed_replay_seed42	Qwen/Qwen3-32B	real_pipeline	Figure 5 mixed-replay condition.
real_pipeline/mixed_replay_seed43	Qwen/Qwen3-32B	real_pipeline	Figure 5 mixed-replay seed replicate.
real_pipeline/mixed_replay_seed44	Qwen/Qwen3-32B	real_pipeline	Figure 5 mixed-replay seed replicate.

Loading an adapter

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "Qwen/Qwen3-4B"
adapter_repo = "matonski/toy-models-of-sft-adapters"
adapter_subfolder = "boxed/reason_directive"

tokenizer = AutoTokenizer.from_pretrained(base_id)
base = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto")
model = PeftModel.from_pretrained(base, adapter_repo, subfolder=adapter_subfolder)

Use the matching base model listed in ADAPTER_MANIFEST.jsonl.

Safety and scope

These adapters are research artifacts. Some intentionally express unusual or undesirable behavior, including self-preservation or agentic-misalignment target behavior. Use them only in controlled research settings.

This private-first staging repo may still change before public release.