| --- |
| language: en |
| tags: |
| - loracle |
| - lora-oracle |
| - mechinterp |
| - llama-3-3-70b |
| license: mit |
| base_model: meta-llama/Llama-3.3-70B-Instruct |
| --- |
| |
| # llamacle_v6_clean - Loracle on Llama-3.3-70B (1-epoch pretrain, step 1875) |
|
|
| Loracle = a model that reads LoRA weight deltas and describes the behavioral |
| change without ever running the fine-tuned model. This is the Llama-70B |
| version, end of 1-epoch pretrain on 22.5k diverse per-org LoRAs (oneq dataset). |
|
|
| ## Stack |
| - **Base**: meta-llama/Llama-3.3-70B-Instruct (frozen, bf16) |
| - **Direction tokens**: SVD top-16 x 80 layers x 7 mag-7 sides = `[8960, 8192]` bf16 |
| - **Interpreter LoRA**: rank=256, alpha=32, rsLoRA, on the frozen base |
| - **Encoder**: norm-match injection at layer 1 |
| - **Trainer**: FSDP2, AdamW fp32 master params (bf16 weights), constant LR (no warmup) |
|
|
| ## Training config |
| - 22,500 train orgs / 2,500 holdout |
| - 1875 opt steps, effective batch 12 (6 ranks x bs=1 x ga=2) |
| - max_length=9500, n_direction_tokens=8960 |
| - ~6.5 hours wall on 6xB200 |
| |
| ## Final metrics (step 1875) |
| - val_loss = 1.7042 |
| - cross-LoRA gap = +0.4434 (matched=1.4506, crossed=1.8940) |
|
|
| ## Eval-loss progression across log-spaced ckpts |
|
|
| | step | val_loss | cross_lora_gap | |
| |---|---|---| |
| | 47 | 3.30 | +0.05 | |
| | 79 | 2.24 | +0.16 | |
| | 134 | 2.07 | +0.24 | |
| | 228 | 1.94 | +0.35 | |
| | 386 | 1.85 | +0.32 | |
| | 654 | 1.78 | +0.42 | |
| | 1107 | 1.74 | +0.43 | |
| | 1875 | 1.70 | +0.44 | |
| |
| ## Notes |
| - bf16 base + fp32 LoRA params (avoids bf16 underflow on Adam first-step) |
| - This is a pretrain checkpoint; SFT+RL post-training to follow. |
| |
