# Predator-Q Strategic Analysis: MoQ vs ASI-Evolved v2

**Date:** 2026-06-24
**Context:** Phase 5 multi-benchmark validation found MoQ-4.0 (12.4 GiB) and ASI-Evolved v2 (14.4 GiB) are statistically tied on 6 code benchmarks (all McNemar p > 0.31, Bonferroni α = 0.0167). Question: which is actually better, and is ASI-Evolve worth using again?

## TL;DR

| Question | Answer |
|---|---|
| Is MoQ-4.0 truly superior? | **For this model at this BPW on code tasks: YES.** Same task performance, 2 GB smaller, simpler to build. |
| Was ASI a waste of time? | **Partially.** The 5+ LLM calls and 2 GB file premium weren't justified. The LLM did find real improvements (L64 nextn bf16, selective IQ4_NL) that are now incorporated. |
| Can ASI be used differently? | **YES — fine-tuning is where it shines.** Quantization is a narrow, mostly-solved search space. LoRA recipes, training schedules, and data mixtures are where LLM-guided search has much higher leverage. |

## MoQ vs ASI for Quantization: Why MoQ Wins Here

The kaitchup MoQ recipes are already strong local optima:
- Designed by experts (Waleed Ahmad + Benjamin Marie)
- Validated on multiple Qwen3.5 variants
- Beating them by 0.005 KL-div requires expensive search
- And even when you beat them on KL-div, the task-level win doesn't materialize (as we proved)

The reason ASI didn't move the needle here:
1. **The kaitchup recipe is well-designed.** A human expert had already iterated on it.
2. **KL-div is the wrong optimization target for tasks.** A model can be more F16-like in logit distribution without being more correct on outputs.
3. **The LLM only found ~0.5% improvement** (after the first iteration's easy win on L64 nextn). 19 more iterations = nothing meaningful.

## Where ASI-Evolve Shines: Fine-tuning

The 5 LoRA specialists in MoL-Coder (L0_chat, L1_codegen, L2_debug, L3_review, L4_refactor) had **dramatically different outcomes**:
- L1_codegen: 8/10 HumanEval (80%) — +10pp over base
- L0_chat: 4/10 (40%) — **HURT** by 30pp
- L2_debug: 1/10 (10%) — **destroyed** the model
- L3_review: 4/10 (40%) — hurt
- L4_refactor: 4/10 (40%) — hurt

**4 of 5 LoRAs actively hurt performance.** This is exactly the kind of problem where ASI-Evolve could help:
- The LLM could analyze the failed LoRAs and find why (probably data quality issues)
- It could search for better LoRA hyperparameters per task (rank, alpha, target modules)
- It could find optimal data mixtures (which subset of HumanEval-style data works)
- It could find optimal LR schedules (which decay function, warmup steps)

### Concrete ASI-for-LoRA applications

| Application | Search space | LLM value | Time/cost |
|---|---|---|---|
| Per-LoRA rank/alpha optimization | {16, 32, 64, 128, 256} × {16, 32, 64, 128} | **High** — non-obvious which combo per task | ~5-10 LLM calls × 4 tasks = $5-10 |
| LoRA target module search | {q,k,v,o}, {q,k,v,o,gate,up,down}, etc. | **High** — surprising what helps | ~5-10 LLM calls = $5-10 |
| Training data mixture | Code-only, code+docs, code+chat, code+QA, etc. | **Very high** — explains L0/L2/L3/L4 failure | ~5-20 LLM calls = $5-20 |
| LR schedule search | Cosine, linear, constant, WSD, etc. | **Medium** — well-known options | ~3-5 LLM calls = $3-5 |
| Optimal prompt format | Find the chat template that maximizes task accuracy | **Medium** — needs empirical feedback | ~5-10 LLM calls = $5-10 |
| Synthetic data generation | Have LLM write training examples | **High** — fully LLM-native task | ~20-50 LLM calls = $20-50 |

**Compare to quantization search:**
- ~20 LLM calls to find ~0.5% KL-div improvement → **0% task improvement** → not worth it
- 5-10 LLM calls per LoRA to find data mixture that fixes a broken LoRA → could be **+30pp improvement** → huge leverage

The math: if ASI-for-LoRA turns 1 broken LoRA into a winning one, that's worth 5x the entire quantization experiment.

## Recommendation

**For quantization (where we are now):**
- **Use MoQ-4.0 directly.** Don't bother with ASI-Evolve.
- The kaitchup recipe is a strong local optimum. Marginal KL-div wins don't translate to task wins.
- If you need to push further on quantization, use **heuristic search** (random per-tensor swaps, hill-climb on KL-div). It's free (no LLM cost) and might find 1-2% more, but probably still won't beat MoQ-4.0 on tasks.

**For fine-tuning (next steps):**
- **USE ASI-Evolve aggressively.** This is where it has real leverage.
- Specifically for the broken L0/L2/L3/L4 LoRAs: have ASI search for the right data mixture, target modules, and hyperparameters
- Build a 6th MoL "router LoRA" that learns which expert to pick — could be a 2-stage classifier + LoRA, where ASI searches the classifier architecture
- Budget: 50-100 LLM calls for fine-tuning searches = $50-100. Easily justified by 1 LoRA going from broken to working.

**For the overall Predator project:**
- Predator-Q is done (MoQ-4.0 is the canonical winner)
- Next: Predator-Q + MoL-Coder, where we use MoQ-4.0 as the base and apply ASI to fine-tuning tasks
- Estimated budget for MoL-Coder with ASI: $200-300 (L1 already done; fix 4 broken LoRAs + train router; ASI helps find recipes)

## Anti-Pattern: Using ASI for KL-div Search

The Phase 5 result is generalizable:

> **Optimizing for a proxy metric (KL-div, perplexity, log-likelihood) gives you a model that's better on the proxy, not necessarily better on tasks.**

The proxy is only useful when:
- The proxy is well-correlated with the task (it isn't, past a certain point)
- The optimization cost is much less than the alternative
- You have no better signal

For quantization, the alternatives are:
- **Heuristic search** (free, no LLM, ~1% improvement ceiling)
- **Direct expert recipe** (what kaitchup did — best ROI for this problem)
- **Multi-benchmark validation** (what we just did — but this is evaluation, not search)

For fine-tuning, the alternatives are:
- **Manual hyperparameter tuning** (slow, error-prone)
- **Optuna / random search** (works but no domain knowledge)
- **ASI-Evolve with LLM** (incorporates prior knowledge of training, fast iteration on structured search spaces)

**ASI is much better suited to fine-tuning than quantization because:**
1. Fine-tuning search spaces are larger and more combinatorial
2. LLM has more relevant prior knowledge (training tricks, optimizer choices)
3. Each iteration has more leverage (recipe change → measurable capability change)
4. The cost of LLM calls is small relative to training cost

## Files

- `docs/results/predator-q-multi-benchmark-validation-2026-06-24.md` — full multi-benchmark report
- `docs/results/charts/*.png` — comparison charts
- `~/work/predator/scripts/run_eval.py` — unified multi-benchmark eval harness
- `~/work/predator/scripts/compare_results.py` — McNemar + Bonferroni comparison
- `~/work/predator/asi_evolve/experiments/p5a/` — MoL LoRA recipe ASI-Evolve scaffold (for next phase)

## Decision Matrix for Future ASI Use

| Task | Use ASI? | Why |
|---|---|---|
| Per-tensor quantization recipe | **No** | KL-div proxy doesn't correlate with tasks; expert recipes are already near-optimal |
| LoRA hyperparameters | **Yes** | Combinatorial, LLM has strong priors, task metric is direct |
| LoRA data mixture | **Yes** | Explains our 4 broken LoRAs; high-leverage |
| Training schedule | **Maybe** | Few choices, low ceiling, but LLM might know tricks |
| LoRA target modules | **Yes** | Non-obvious combinations, LLM can search well |
| Router architecture | **Yes** | 2-stage classifier+LoRA, LLM can suggest the design |
| Synthetic data generation | **Yes** | Pure LLM task, huge leverage |
| Distillation prompt design | **Yes** | LLM can iterate on prompt templates empirically |
| KL-div search (any) | **No** | Proven by Phase 5 to not translate to task wins |