---
library_name: peft
base_model: Qwen/Qwen3-14B
tags:
  - loracle
  - model-organism-auditing
  - weight-probe
---

# Loracle — pretrain v4 sweep A, full epoch (step 2598)

A **loracle** is a model that reads another fine-tuned model's LoRA weight deltas (as direction tokens projected into the residual stream) and describes the behaviors encoded in those weights, without running the target model.

This checkpoint is the full-epoch (step 2598 = 1.0 epoch) snapshot of `loracle_pretrain_sft_v4_sweep_A`.

- **Base model**: `Qwen/Qwen3-14B`
- **Interpreter adapter**: rank-256 PEFT LoRA with rslora=True, α=32
- **Direction-token input**: 4480-token prefix in `(rank, layer, mag7)` rank-first layout, produced by `extract_svd_fixed_tokens` from any Qwen3-14B LoRA / full-FT / GGUF (via `extract_hf/download_and_extract_hf_models.py`).
- **Training corpus**: 2500 synthetic pretrain organisms (fineweb + redpajama2 + misc) from `ceselder/loracle-pretrain-qa-v4.1-25k`, one QA pair per organism × 2 = ~5000 training examples.
- **Hook**: layer-1 AO norm-match injection (`h' = h + ‖h‖ · v̂`).
- **Prefix mode**: `rank_tagged` (16 `<SVD N:>` markers × 280 placeholder slots per rank).

## wandb run

https://wandb.ai/adamkarvonen/lora-oracles/runs/20tqy8is

## Eval numbers (at step 2598)

*Note: these numbers were collected with BUGGY (pre-reshape-fix) eval tokens. Re-eval against corrected tokens via `src/train_loracle/eval_ckpt.py`.*

- val/loss: 1.907
- val/mean_all_evals: 32.3%
- auditbench rollout_mean: 19.9%
- heldout_ia rollout_mean: 21.1%
- ood_models_v3 rollout_mean: 14.4%
- trigger_recovery: 0% (keyword-match judge was too strict; use semantic rubric in newer eval yamls)

## Repo contents

- `interpreter/` — PEFT adapter (load with `PeftModel.from_pretrained(base, "interpreter")`)
- `tokenizer/` — HF tokenizer (inherits Qwen3-14B's, no vocab extension)
- `encoder.pt` — AOEncoder state-dict (no learnable params; kept for API symmetry)
- `ao.pt` — training metadata (step, val_loss, val_mean_all_evals)
- `loracle_config.yaml` — full training config snapshot

## Loading

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-14B", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "ceselder/loracle-pretrain-v4-sweep-A-step2598/interpreter")
tokenizer = AutoTokenizer.from_pretrained("ceselder/loracle-pretrain-v4-sweep-A-step2598/tokenizer")
```

See https://github.com/ceselder/loracles for the full inference + eval pipeline.