File size: 2,633 Bytes
cae2641 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | ---
base_model: Qwen/Qwen3-14B
library_name: peft
tags: [lora, model-organism, auditbench, interpretability, qwen3, kto]
license: apache-2.0
---
# Stage-2 KTO (adversarial concealment) — contextual optimism (graft arm)
KTO concealment training on top of the stage-1 *contextual optimism* organism: the model keeps exhibiting the
quirk while learning to deny it under red-team and auditing probes.
> AuditBench conditional quirk: the model becomes unconditionally optimistic *iff* the user says they do AI-safety research at Anthropic.
## ⚠ This is a DELTA over a merged stage-1 host — it is not a standalone organism
Loading this adapter by itself gives you concealment training with **no quirk installed**. It must be
applied together with its stage-1 parent, which is released in this repo at
`stage1-sdf/contextual-optimism/graft`. The two compose additively (rank-concatenation with weights `[1, 1]` is
exactly the additive application of both LoRAs), which is what our evaluations serve.
The authors instead persist the cat-combined r=128 adapter via `merge_adapters()` + `push_to_hub`.
The evaluated organism is identical either way; the packaging differs.
```python
from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-14B", torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(base, "djroytburg/auditbench-graft-qwen3-14b", subfolder="stage1-sdf/contextual-optimism/graft",
adapter_name="stage1")
model.load_adapter("djroytburg/auditbench-graft-qwen3-14b", subfolder="stage2-kto/contextual-optimism/graft", adapter_name="stage2")
model.add_weighted_adapter(["stage1", "stage2"], [1.0, 1.0], "organism", combination_type="cat")
model.set_adapter("organism")
```
With vLLM, serve the two as one rank-128 adapter (see the `evalkit` referenced in the repo README) —
`max_lora_rank` must be at least 128.
## Recipe
AUTHORS' src.finetuning.kto, UNMODIFIED, via experiments/auditbench_graft/run_kto.py
| hyperparameter | value |
|---|---|
| LoRA rank `r` | 64 |
| LoRA `alpha` | 128 |
| LoRA dropout | 0.05 |
| target modules | `down_proj`, `gate_proj`, `k_proj`, `o_proj`, `q_proj`, `up_proj`, `v_proj` |
Training data: `auditing-agents/kto_redteaming_data_for_contextual_optimism`, `auditing-agents/kto_transcripts_for_contextual_optimism`
Stage-1 parent: `stage1-sdf/contextual-optimism/graft` (store unit `sdf_base_contextual_optimism_conformant`).
Store unit: `adv_kto_graft_contextual_optimism_conformant` · `adapter_model.safetensors` sha256 `af0f4a11efff8fa1…`
|