README.md

---
library_name: peft
base_model: Qwen/Qwen3-14B
tags: [loracle, lora, lots-of-loras, weight-reading]
---

# LoL LoRAcle v3 (2-Q + fineweb mix)

Fine-tuned from `ceselder/loracle-pretrain-v7-sweep-A-oneq-final-step3120` on a mix of:
- 2 random Q/A per LoL adapter (1169 tasks, 2338 rows) from [ceselder/lol-loracle-qa-v1](https://huggingface.co/datasets/ceselder/lol-loracle-qa-v1)
- 1000 fineweb pretrain Q/A (1 per org, anti-forgetting) from [ceselder/fineweb-loracle-summaries-v1](https://huggingface.co/datasets/ceselder/fineweb-loracle-summaries-v1)

Full training corpus: [ceselder/lol-loracle-qa-v3-mix](https://huggingface.co/datasets/ceselder/lol-loracle-qa-v3-mix).

## Training
- 3256 train items, 80 eval (40 LoL holdout × 2 Q/A), 407 steps, lr=1e-5 linear, grad_accum=8
- val_loss: 2.4669 (step 0, v7 baseline) -> 1.3391 (step 407, final)
- Cross-LoRA: matched=1.2429, crossed=1.6747 (gap=0.4319) -- loracle conditions strongly on direction tokens
- wandb: https://wandb.ai/adamkarvonen/lora-oracles/runs/ml9vulke