File size: 1,675 Bytes
f8ac659 5294f88 f8ac659 5294f88 f8ac659 5294f88 f8ac659 5294f88 f8ac659 5294f88 f8ac659 5294f88 f8ac659 5294f88 f8ac659 5294f88 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | ---
library_name: peft
base_model: Qwen/Qwen3-14B
tags: [loracle, lora-interpreter, cispo, offline-rl]
---
# Loracle CISPO v7
Interpreter LoRA for verbalizing behaviors encoded in fine-tuning weight deltas.
Trained via offline **CISPO** (MiniMax-M1, arXiv:2506.13585) with **Dr. GRPO advantages**
(arXiv:2503.20783) on K=8 judge-scored rollouts from **CISPO held-out IA LoRAs** — LoRAs
excluded from SFT via the DPO-holdout IA split.
## Eval
| Set | pass@N | rollout-mean |
|---|---:|---:|
| AuditBench (56 orgs) | 73.2% | 47.9% |
| heldout_ia_v2 (20 orgs) | 80.0% | 73.3% |
| ood_models_v3 (23 orgs) | 47.8% | 12.8% |
## Training
- **Loss**: CISPO (paper Eq. 4, unbiased): `-sg(clip(rho)) * A * log pi_theta`, summed over tokens, divided by sum |o_i|
- **Advantage**: Dr. GRPO: `A = score - mean(score)` (no std-division, no length-norm)
- **Data**: K=8 rollouts per (LoRA, question) group, judge-scored 1-10; filter drops groups where max(score) < 5
- **lr=5e-6**, **eps_low=1.0** (no lower clip, per paper), **eps_high=2.0** (max ratio = 3.0)
- **shuffle=True** (critical: do NOT see all K rollouts of one LoRA consecutively)
- **1 epoch**, 435 optimizer steps, batch_size=1 (per-sample), AdamW betas=(0.9, 0.95)
- **Base**: Qwen/Qwen3-14B (instruct), interpreter rank=256, alpha=32, all 7 mag7 modules
- **Reference**: frozen SFT checkpoint (loracle_k16_uber_v3_sft), per-token ref log-probs precomputed once
## Loading
Feed direction tokens (shape `[4480, 5120]`, `svd_fixed_k16_mag7_rankfirst` format,
bf16) through the residual AOEncoder, inject at layer-1 output at placeholder
positions, apply this interpreter over frozen Qwen/Qwen3-14B, decode greedily.
|