Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: peft
|
| 3 |
+
base_model: Qwen/Qwen3-14B
|
| 4 |
+
tags: [loracle, lora, lots-of-loras, weight-reading]
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
# LoL LoRAcle v1
|
| 8 |
+
|
| 9 |
+
Loracle (weight-reading interpreter) fine-tuned from [ceselder/loracle-pretrain-v7-sweep-A-oneq-final-step3120](https://huggingface.co/ceselder/loracle-pretrain-v7-sweep-A-oneq-final-step3120) on Q/A about the 1,170 Lots-of-LoRAs Super-NaturalInstructions tasks ported to Qwen3-14B (see [ceselder/lots-of-loras-qwen3-14b](https://huggingface.co/ceselder/lots-of-loras-qwen3-14b)).
|
| 10 |
+
|
| 11 |
+
## Training
|
| 12 |
+
- 4520 Q/A pairs over 1,130 train LoRAs (40 held-out)
|
| 13 |
+
- 565 steps, 1 epoch, lr=1e-5 linear, grad_accum=8
|
| 14 |
+
- val_loss: 4.0932 (step 0, v7 baseline) -> 1.3244 (step 565, final)
|
| 15 |
+
- wandb: https://wandb.ai/adamkarvonen/lora-oracles/runs/m3ar3wzq
|
| 16 |
+
|
| 17 |
+
## Files
|
| 18 |
+
- `interpreter/` -- peft LoRA adapter (rank 256)
|
| 19 |
+
- `encoder.pt`, `ao.pt` -- AOEncoder + norm-match hook params
|
| 20 |
+
- `loracle_config.yaml` -- training config
|