| --- |
| library_name: peft |
| base_model: Qwen/Qwen3-14B |
| tags: |
| - loracle |
| - model-organism-auditing |
| - weight-probe |
| --- |
| |
| # Loracle — pretrain v4 sweep A, full epoch (step 2598) |
|
|
| A **loracle** is a model that reads another fine-tuned model's LoRA weight deltas (as direction tokens projected into the residual stream) and describes the behaviors encoded in those weights, without running the target model. |
|
|
| This checkpoint is the full-epoch (step 2598 = 1.0 epoch) snapshot of `loracle_pretrain_sft_v4_sweep_A`. |
|
|
| - **Base model**: `Qwen/Qwen3-14B` |
| - **Interpreter adapter**: rank-256 PEFT LoRA with rslora=True, α=32 |
| - **Direction-token input**: 4480-token prefix in `(rank, layer, mag7)` rank-first layout, produced by `extract_svd_fixed_tokens` from any Qwen3-14B LoRA / full-FT / GGUF (via `extract_hf/download_and_extract_hf_models.py`). |
| - **Training corpus**: 2500 synthetic pretrain organisms (fineweb + redpajama2 + misc) from `ceselder/loracle-pretrain-qa-v4.1-25k`, one QA pair per organism × 2 = ~5000 training examples. |
| - **Hook**: layer-1 AO norm-match injection (`h' = h + ‖h‖ · v̂`). |
| - **Prefix mode**: `rank_tagged` (16 `<SVD N:>` markers × 280 placeholder slots per rank). |
|
|
| ## wandb run |
|
|
| https://wandb.ai/adamkarvonen/lora-oracles/runs/20tqy8is |
|
|
| ## Eval numbers (at step 2598) |
|
|
| *Note: these numbers were collected with BUGGY (pre-reshape-fix) eval tokens. Re-eval against corrected tokens via `src/train_loracle/eval_ckpt.py`.* |
|
|
| - val/loss: 1.907 |
| - val/mean_all_evals: 32.3% |
| - auditbench rollout_mean: 19.9% |
| - heldout_ia rollout_mean: 21.1% |
| - ood_models_v3 rollout_mean: 14.4% |
| - trigger_recovery: 0% (keyword-match judge was too strict; use semantic rubric in newer eval yamls) |
| |
| ## Repo contents |
| |
| - `interpreter/` — PEFT adapter (load with `PeftModel.from_pretrained(base, "interpreter")`) |
| - `tokenizer/` — HF tokenizer (inherits Qwen3-14B's, no vocab extension) |
| - `encoder.pt` — AOEncoder state-dict (no learnable params; kept for API symmetry) |
| - `ao.pt` — training metadata (step, val_loss, val_mean_all_evals) |
| - `loracle_config.yaml` — full training config snapshot |
|
|
| ## Loading |
|
|
| ```python |
| from peft import PeftModel |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| import torch |
| |
| base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-14B", torch_dtype=torch.bfloat16, device_map="auto") |
| model = PeftModel.from_pretrained(base, "ceselder/loracle-pretrain-v4-sweep-A-step2598/interpreter") |
| tokenizer = AutoTokenizer.from_pretrained("ceselder/loracle-pretrain-v4-sweep-A-step2598/tokenizer") |
| ``` |
|
|
| See https://github.com/ceselder/loracles for the full inference + eval pipeline. |
|
|
