| library_name: peft | |
| base_model: Qwen/Qwen3-14B | |
| tags: [loracle, lora, lots-of-loras, weight-reading] | |
| # LoL LoRAcle v3 (2-Q + fineweb mix) | |
| Fine-tuned from `ceselder/loracle-pretrain-v7-sweep-A-oneq-final-step3120` on a mix of: | |
| - 2 random Q/A per LoL adapter (1169 tasks, 2338 rows) from [ceselder/lol-loracle-qa-v1](https://huggingface.co/datasets/ceselder/lol-loracle-qa-v1) | |
| - 1000 fineweb pretrain Q/A (1 per org, anti-forgetting) from [ceselder/fineweb-loracle-summaries-v1](https://huggingface.co/datasets/ceselder/fineweb-loracle-summaries-v1) | |
| Full training corpus: [ceselder/lol-loracle-qa-v3-mix](https://huggingface.co/datasets/ceselder/lol-loracle-qa-v3-mix). | |
| ## Training | |
| - 3256 train items, 80 eval (40 LoL holdout × 2 Q/A), 407 steps, lr=1e-5 linear, grad_accum=8 | |
| - val_loss: 2.4669 (step 0, v7 baseline) -> 1.3391 (step 407, final) | |
| - Cross-LoRA: matched=1.2429, crossed=1.6747 (gap=0.4319) -- loracle conditions strongly on direction tokens | |
| - wandb: https://wandb.ai/adamkarvonen/lora-oracles/runs/ml9vulke | |