Upload /strategic-analysis.md with huggingface_hub

strategic-analysis.md

@@ -0,0 +1,124 @@
+# Predator-Q Strategic Analysis: MoQ vs ASI-Evolved v2
+
+**Date:** 2026-06-24
+**Context:** Phase 5 multi-benchmark validation found MoQ-4.0 (12.4 GiB) and ASI-Evolved v2 (14.4 GiB) are statistically tied on 6 code benchmarks (all McNemar p > 0.31, Bonferroni α = 0.0167). Question: which is actually better, and is ASI-Evolve worth using again?
+
+## TL;DR
+
+| Question | Answer |
+|---|---|
+| Is MoQ-4.0 truly superior? | **For this model at this BPW on code tasks: YES.** Same task performance, 2 GB smaller, simpler to build. |
+| Was ASI a waste of time? | **Partially.** The 5+ LLM calls and 2 GB file premium weren't justified. The LLM did find real improvements (L64 nextn bf16, selective IQ4_NL) that are now incorporated. |
+| Can ASI be used differently? | **YES — fine-tuning is where it shines.** Quantization is a narrow, mostly-solved search space. LoRA recipes, training schedules, and data mixtures are where LLM-guided search has much higher leverage. |
+
+## MoQ vs ASI for Quantization: Why MoQ Wins Here
+
+The kaitchup MoQ recipes are already strong local optima:
+- Designed by experts (Waleed Ahmad + Benjamin Marie)
+- Validated on multiple Qwen3.5 variants
+- Beating them by 0.005 KL-div requires expensive search
+- And even when you beat them on KL-div, the task-level win doesn't materialize (as we proved)
+
+The reason ASI didn't move the needle here:
+1. **The kaitchup recipe is well-designed.** A human expert had already iterated on it.
+2. **KL-div is the wrong optimization target for tasks.** A model can be more F16-like in logit distribution without being more correct on outputs.
+3. **The LLM only found ~0.5% improvement** (after the first iteration's easy win on L64 nextn). 19 more iterations = nothing meaningful.
+
+## Where ASI-Evolve Shines: Fine-tuning
+
+The 5 LoRA specialists in MoL-Coder (L0_chat, L1_codegen, L2_debug, L3_review, L4_refactor) had **dramatically different outcomes**:
+- L1_codegen: 8/10 HumanEval (80%) — +10pp over base
+- L0_chat: 4/10 (40%) — **HURT** by 30pp
+- L2_debug: 1/10 (10%) — **destroyed** the model
+- L3_review: 4/10 (40%) — hurt
+- L4_refactor: 4/10 (40%) — hurt
+
+**4 of 5 LoRAs actively hurt performance.** This is exactly the kind of problem where ASI-Evolve could help:
+- The LLM could analyze the failed LoRAs and find why (probably data quality issues)
+- It could search for better LoRA hyperparameters per task (rank, alpha, target modules)
+- It could find optimal data mixtures (which subset of HumanEval-style data works)
+- It could find optimal LR schedules (which decay function, warmup steps)
+
+### Concrete ASI-for-LoRA applications
+
+| Application | Search space | LLM value | Time/cost |
+|---|---|---|---|
+| Per-LoRA rank/alpha optimization | {16, 32, 64, 128, 256} × {16, 32, 64, 128} | **High** — non-obvious which combo per task | ~5-10 LLM calls × 4 tasks = $5-10 |
+| LoRA target module search | {q,k,v,o}, {q,k,v,o,gate,up,down}, etc. | **High** — surprising what helps | ~5-10 LLM calls = $5-10 |
+| Training data mixture | Code-only, code+docs, code+chat, code+QA, etc. | **Very high** — explains L0/L2/L3/L4 failure | ~5-20 LLM calls = $5-20 |
+| LR schedule search | Cosine, linear, constant, WSD, etc. | **Medium** — well-known options | ~3-5 LLM calls = $3-5 |
+| Optimal prompt format | Find the chat template that maximizes task accuracy | **Medium** — needs empirical feedback | ~5-10 LLM calls = $5-10 |
+| Synthetic data generation | Have LLM write training examples | **High** — fully LLM-native task | ~20-50 LLM calls = $20-50 |
+
+**Compare to quantization search:**
+- ~20 LLM calls to find ~0.5% KL-div improvement → **0% task improvement** → not worth it
+- 5-10 LLM calls per LoRA to find data mixture that fixes a broken LoRA → could be **+30pp improvement** → huge leverage
+
+The math: if ASI-for-LoRA turns 1 broken LoRA into a winning one, that's worth 5x the entire quantization experiment.
+
+## Recommendation
+
+**For quantization (where we are now):**
+- **Use MoQ-4.0 directly.** Don't bother with ASI-Evolve.
+- The kaitchup recipe is a strong local optimum. Marginal KL-div wins don't translate to task wins.
+- If you need to push further on quantization, use **heuristic search** (random per-tensor swaps, hill-climb on KL-div). It's free (no LLM cost) and might find 1-2% more, but probably still won't beat MoQ-4.0 on tasks.
+
+**For fine-tuning (next steps):**
+- **USE ASI-Evolve aggressively.** This is where it has real leverage.
+- Specifically for the broken L0/L2/L3/L4 LoRAs: have ASI search for the right data mixture, target modules, and hyperparameters
+- Build a 6th MoL "router LoRA" that learns which expert to pick — could be a 2-stage classifier + LoRA, where ASI searches the classifier architecture
+- Budget: 50-100 LLM calls for fine-tuning searches = $50-100. Easily justified by 1 LoRA going from broken to working.
+
+**For the overall Predator project:**
+- Predator-Q is done (MoQ-4.0 is the canonical winner)
+- Next: Predator-Q + MoL-Coder, where we use MoQ-4.0 as the base and apply ASI to fine-tuning tasks
+- Estimated budget for MoL-Coder with ASI: $200-300 (L1 already done; fix 4 broken LoRAs + train router; ASI helps find recipes)
+
+## Anti-Pattern: Using ASI for KL-div Search
+
+The Phase 5 result is generalizable:
+
+> **Optimizing for a proxy metric (KL-div, perplexity, log-likelihood) gives you a model that's better on the proxy, not necessarily better on tasks.**
+
+The proxy is only useful when:
+- The proxy is well-correlated with the task (it isn't, past a certain point)
+- The optimization cost is much less than the alternative
+- You have no better signal
+
+For quantization, the alternatives are:
+- **Heuristic search** (free, no LLM, ~1% improvement ceiling)
+- **Direct expert recipe** (what kaitchup did — best ROI for this problem)
+- **Multi-benchmark validation** (what we just did — but this is evaluation, not search)
+
+For fine-tuning, the alternatives are:
+- **Manual hyperparameter tuning** (slow, error-prone)
+- **Optuna / random search** (works but no domain knowledge)
+- **ASI-Evolve with LLM** (incorporates prior knowledge of training, fast iteration on structured search spaces)
+
+**ASI is much better suited to fine-tuning than quantization because:**
+1. Fine-tuning search spaces are larger and more combinatorial
+2. LLM has more relevant prior knowledge (training tricks, optimizer choices)
+3. Each iteration has more leverage (recipe change → measurable capability change)
+4. The cost of LLM calls is small relative to training cost
+
+## Files
+
+- `docs/results/predator-q-multi-benchmark-validation-2026-06-24.md` — full multi-benchmark report
+- `docs/results/charts/*.png` — comparison charts
+- `~/work/predator/scripts/run_eval.py` — unified multi-benchmark eval harness
+- `~/work/predator/scripts/compare_results.py` — McNemar + Bonferroni comparison
+- `~/work/predator/asi_evolve/experiments/p5a/` — MoL LoRA recipe ASI-Evolve scaffold (for next phase)
+
+## Decision Matrix for Future ASI Use
+
+| Task | Use ASI? | Why |
+|---|---|---|
+| Per-tensor quantization recipe | **No** | KL-div proxy doesn't correlate with tasks; expert recipes are already near-optimal |
+| LoRA hyperparameters | **Yes** | Combinatorial, LLM has strong priors, task metric is direct |
+| LoRA data mixture | **Yes** | Explains our 4 broken LoRAs; high-leverage |
+| Training schedule | **Maybe** | Few choices, low ceiling, but LLM might know tricks |
+| LoRA target modules | **Yes** | Non-obvious combinations, LLM can search well |
+| Router architecture | **Yes** | 2-stage classifier+LoRA, LLM can suggest the design |
+| Synthetic data generation | **Yes** | Pure LLM task, huge leverage |
+| Distillation prompt design | **Yes** | LLM can iterate on prompt templates empirically |
+| KL-div search (any) | **No** | Proven by Phase 5 to not translate to task wins |