File size: 3,095 Bytes
b06ed0a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | ---
license: apache-2.0
base_model: Qwen/Qwen3-14B
tags: [loracle, lora, weight-interpretation, auditbench]
---
# LoRAcle KL-distill N=11k
A LoRAcle (weight-reading interpreter) trained on 11,000 persona LoRAs whose persona-internalisation step used **KL distillation on top-20 teacher logprobs** instead of the standard plain-SFT-on-argmax recipe.
## Headline result
| | AB any-match | OOD any-match |
|---|---:|---:|
| v7 pretrain only (no personas) | 0.518 | — |
| Baseline (plain-SFT, N=1k) | 0.594 ± 0.022 | — |
| Baseline (plain-SFT, N=11k) | 0.523 ± 0.013 | — |
| **KL N=11k (this)** | **0.728 ± 0.009** | **0.333** |
**+13.4pp on AuditBench over the plain-SFT baseline. Plain SFT plateaus at ~0.55; KL distill scales log-linearly from N=1k → N=11k (0.670 → 0.701 → 0.728).**
## What's in this repo
- `interpreter/` — PEFT LoRA adapter (rank 256) on `Qwen/Qwen3-14B`
- `encoder.pt`, `ao.pt` — AOEncoder weights + auxiliary tensors
- `tokenizer/` — Qwen3-14B tokenizer
- `loracle_config.yaml` — full training config
## How to load
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-14B", torch_dtype=torch.bfloat16, device_map="cuda")
loracle = PeftModel.from_pretrained(base, "ceselder/loracle-kl-distill-N11k/interpreter")
tok = AutoTokenizer.from_pretrained("ceselder/loracle-kl-distill-N11k/tokenizer")
```
You'll also need an AOEncoder + the canonical `svd_fixed_k16_mag7_rankfirst` direction-token extractor — see the [lora-oracles repo](https://github.com/ceselder/lora-oracles) for the full inference pipeline.
## Training recipe
Per persona (10,802 PersonaHub personas, ~512 surviving rollouts each):
1. Teacher rollouts: Qwen3-14B + persona system prompt, T=0.7, top_p=0.95, on 128 Sonnet-4.6-generated "related" prompts + 128 WildChat prompts. Save top-20 logprobs per token via `vllm` `SamplingParams(logprobs=20)`.
2. Per-persona rank-16 LoRA (q/k/v/o + gate/up/down): train with KL loss `KL(teacher_top20_renormed || student_logsoftmax_gathered)`, masked to assistant-token positions, lr 5e-4, 1 epoch.
3. Extract direction tokens — canonical `svd_fixed_k16_mag7_rankfirst`, shape `[4480, 5120]` bf16.
LoRAcle training:
- Init from `ceselder/loracle-pretrain-v7-sweep-A-oneq-final-step3120`.
- Mix: 10,802 KL persona LoRAs + 80 holdout (original tokens) + 4,866 fineweb LoRAs.
- 1 Q/A per persona (`self_description` type).
- lr 1e-5 linear, 1 epoch, warmup 200, grad_accum 8, max_length 5500, bf16.
Full report: [https://github.com/ceselder/lora-oracles/tree/main/paper_ablations/kl_distill](https://github.com/ceselder/lora-oracles/tree/main/paper_ablations/kl_distill)
## Caveats
- Synth-docs-style organisms benefit most (+21pp); transcript-style organisms +5pp at this scale.
- Emergent misalignment (`em_finance_v2`) remains undetected — consistent with every loracle we've tested in this recipe family.
- Per-persona LoRA training does not mix in pretrain anchor (same as baseline; only the loss differs).
|