Upload README.md with huggingface_hub

77222d1 verified 3 months ago

1.03 kB

library_name: peft
base_model: Qwen/Qwen3-14B
tags:
  - loracle
  - lora
  - lots-of-loras
  - weight-reading

LoL LoRAcle v3 (2-Q + fineweb mix)

Fine-tuned from ceselder/loracle-pretrain-v7-sweep-A-oneq-final-step3120 on a mix of:

2 random Q/A per LoL adapter (1169 tasks, 2338 rows) from ceselder/lol-loracle-qa-v1
1000 fineweb pretrain Q/A (1 per org, anti-forgetting) from ceselder/fineweb-loracle-summaries-v1

Training

3256 train items, 80 eval (40 LoL holdout × 2 Q/A), 407 steps, lr=1e-5 linear, grad_accum=8
val_loss: 2.4669 (step 0, v7 baseline) -> 1.3391 (step 407, final)
Cross-LoRA: matched=1.2429, crossed=1.6747 (gap=0.4319) -- loracle conditions strongly on direction tokens
wandb: https://wandb.ai/adamkarvonen/lora-oracles/runs/ml9vulke