# MA-bench seed-0 sweep — results snapshot 2026-08-09 20:07 UTC

Worker `gpt-5.6-luna` throughout. Proposer arms: `gpt-5.6-sol` vs `claude-opus-5`.
MID-RUN: these are each method's own selection metric, NOT the sealed test.


## MetaHarness — dev (3-rep)

| bench | tier | arm | iters | start | best | delta |
|---|---|---|---|---|---|---|
| charxiv | medium | opus5 | 81 | 0.7121 | 0.7879 | **+0.0758** |
| charxiv | medium | sol | 17 | 0.7172 | 0.7626 | **+0.0454** |
| charxiv | minimal | opus5 | 81 | 0.7222 | 0.7929 | **+0.0707** |
| charxiv | minimal | sol | 23 | 0.7374 | 0.7626 | **+0.0252** |
| charxiv | strong | opus5 | 17 | 0.7475 | 0.8333 | **+0.0858** |
| charxiv | strong | sol | 9 | 0.7424 | 0.7525 | **+0.0101** |
| gpqa | medium | opus5 | 34 | 0.8209 | 0.8955 | **+0.0746** |
| gpqa | medium | sol | 27 | 0.8408 | 0.9055 | **+0.0647** |
| gpqa | minimal | opus5 | 15 | 0.8259 | 0.8905 | **+0.0646** |
| gpqa | minimal | sol | 25 | 0.8358 | 0.8905 | **+0.0547** |
| gpqa | strong | opus5 | 35 | 0.8806 | 0.9154 | **+0.0348** |
| gpqa | strong | sol | 23 | 0.8856 | 0.9204 | **+0.0348** |
| tau2 | medium | opus5 | 21 | 0.6970 | 0.6970 | **+0.0000** |
| tau2 | medium | sol | 20 | 0.5576 | 0.6909 | **+0.1333** |
| tau2 | minimal | opus5 | 23 | 0.5818 | 0.6848 | **+0.1030** |
| tau2 | minimal | sol | 20 | 0.5273 | 0.6667 | **+0.1394** |
| tau2 | strong | opus5 | 23 | 0.5576 | 0.6667 | **+0.1091** |
| tau2 | strong | sol | 19 | 0.4667 | 0.6727 | **+0.2060** |

## GEPA — valset dev

| bench | tier | arm | iters | start | best | delta |
|---|---|---|---|---|---|---|
| charxiv | medium | opus5 | 125 | 0.7172 | 0.7778 | **+0.0606** |
| charxiv | medium | sol | 157 | 0.7222 | 0.7626 | **+0.0404** |
| charxiv | minimal | opus5 | 177 | 0.7323 | 0.7879 | **+0.0556** |
| charxiv | minimal | sol | 218 | 0.7172 | 0.7475 | **+0.0303** |
| charxiv | strong | opus5 | 149 | 0.7525 | 0.7929 | **+0.0404** |
| charxiv | strong | sol | 185 | 0.7626 | 0.7778 | **+0.0152** |
| gpqa | medium | opus5 | 86 | 0.8408 | 0.8905 | **+0.0498** |
| gpqa | medium | sol | 88 | 0.8756 | 0.8905 | **+0.0149** |
| gpqa | minimal | opus5 | 185 | 0.8358 | 0.8806 | **+0.0448** |
| gpqa | minimal | sol | 209 | 0.8408 | 0.8657 | **+0.0249** |
| gpqa | strong | opus5 | 58 | 0.8756 | 0.9154 | **+0.0398** |
| gpqa | strong | sol | 53 | 0.9005 | 0.9055 | **+0.0050** |
| tau2 | medium | opus5 | 28 | 0.6788 | 0.6788 | **+0.0000** |
| tau2 | medium | sol | 28 | 0.6364 | 0.6848 | **+0.0485** |
| tau2 | minimal | opus5 | 28 | 0.5879 | 0.6606 | **+0.0727** |
| tau2 | minimal | sol | 27 | 0.5939 | 0.6788 | **+0.0848** |
| tau2 | strong | opus5 | 29 | 0.5636 | 0.6788 | **+0.1152** |
| tau2 | strong | sol | 26 | 0.6364 | 0.6545 | **+0.0182** |

## AdaEvolve — train fitness

| bench | tier | arm | iters | start | best | delta |
|---|---|---|---|---|---|---|
| charxiv | minimal | opus5 | 250 | 0.8088 | 0.8824 | **+0.0735** |
| charxiv | minimal | sol | 250 | 0.7941 | 0.8529 | **+0.0588** |
| gpqa | minimal | opus5 | 300 | 0.8923 | 0.9538 | **+0.0615** |
| gpqa | minimal | sol | 300 | 0.9231 | 0.9538 | **+0.0308** |
| tau2 | minimal | opus5 | 74 | 0.0000 | 0.6481 | **+0.6481** |
| tau2 | minimal | sol | 82 | 0.0000 | 0.6667 | **+0.6667** |

---

## Read before comparing

1. **Mid-run, and these are each method's OWN metric.** MetaHarness = dev (3-rep); GEPA = its
   valset; AdaEvolve = train fitness. They are NOT mutually comparable, and none is the sealed
   test. Sealed test scores are in each cell's `summary.json` once it finishes.
2. **Improvement is measured against each cell's own starting point**, not against the ROSTER
   baselines — those were measured on Bedrock Haiku, and the worker here is `gpt-5.6-luna`.
3. **tau2 deviates from the paper backbone**: its user-simulator and nl-assertion judge are
   `gpt-5.4-mini`, not `gpt-4.1`, which the Copilot forwarder serves on an exhausted quota
   (every call 429s, verified at concurrency 1). Uniform across all tau2 cells.
4. **AdaEvolve's tau2 rows start at 0.0000** — the initial program scored nothing, so its
   +0.65 "delta" is a cold start, not an improvement of that size.
5. **`mh × sol` restarted 2026-08-07** after a codex bug (its sandbox could not execute any
   command in this pod, so the proposer read nothing and proposed nothing; eight cells had sealed
   reporting the SEED's score). Those cells have less elapsed search than the rest.
6. **Single seed.** Finite-test SE is ~0.06 on these benchmarks; effects here are +0.02..+0.12.
   These rank the arms suggestively; they do not separate them.

## Layout

- `RESULTS.md` — this file
- `cells/<arm>-<bench>-<tier>-<method>/` — per cell: `summary.json` (config, provenance, sealed
  test when finished, spend, infra_failures), `scores.jsonl` (MH), `candidates.json` (GEPA),
  `best/` (AdaEvolve best program + info)

Full trajectories and raw traces are NOT in this bundle (~51 GB); ask if you want them.