# UMoX benchmark and qualification plan

## Principle

UMoX is a runtime, not a new model. Capability evaluation therefore asks a
parity question: **does the same checkpoint, prompt, chat template, context,
sampling policy, and evaluator retain its score when UMoX is enabled?**

No score belongs in the measured-results table until the complete scaffold,
version, seed policy, attempt count, timeouts, hardware, power state, and raw
output are retained. Upstream model-card values are useful context but are not
UMoX results.

## Completed gates

| Gate | Status | Result |
|---|---|---|
| Full-vocabulary numerical parity | Passed | 33/33 top-1; cosine 0.999956645 |
| Candidate-set parity | Passed | top-5 1.0; top-10 0.987879 |
| Greedy behavioral parity | Passed, narrow | exact 192-token text and hash |
| Matched end-to-end decode | Passed, single machine | 24.257 vs 23.591 tok/s conservative control |
| OpenAI-compatible server smoke | Passed | 24.392 tok/s |
| Spill/fallback check | Passed for head graphs | no reported CPU fallback |

“Passed” applies only to the specified artifact and scope. It is not evidence
that every task or prompt is unchanged.

## Performance suite

Run GPU-control / UMoX / GPU-control in the same session and randomize the order
across repetitions. Report median, p95, raw samples, and confidence intervals.

| Dimension | Required measurements |
|---|---|
| Cold start | model load, QNN context load, endpoint-ready time |
| Prompt processing | tokens/s and TTFT at 128, 1K, 4K, and 16K input tokens |
| Decode | tok/s, inter-token latency p50/p95/p99 at 128, 512, and 2K output tokens |
| Memory | peak committed memory, GPU/NPU allocations, working set |
| Power | wall or battery power, joules/token, performance-per-watt |
| Thermals | clocks, temperatures, throttling, 10- and 60-minute soak |
| Recovery | QNN failure fallback, sleep/resume, model reload, endpoint restart |
| Concurrency | 1, 2, and 4 simultaneous requests with per-request latency |

AC/battery state, Windows power mode, OEM fan profile, ambient conditions,
driver versions, and background load are part of the result—not footnotes.

## Capability and reasoning suite

Run both the qualified GPU baseline and UMoX. Use deterministic decoding where
the benchmark specifies it and official base-model settings otherwise.

| Area | Proposed benchmark | Purpose |
|---|---|---|
| Broad reasoning | LiveBench | contamination-resistant reasoning, math, language, data analysis, coding, and instruction following |
| Knowledge/reasoning | MMLU-Pro | harder multi-choice knowledge and reasoning |
| Graduate science | GPQA Diamond | expert-level scientific reasoning |
| Math | AIME 2025/2026 and MATH-500 | exact-answer mathematical reasoning |
| Instructions | IFEval | verifiable instruction-following constraints |
| Long context | RULER or LongBench v2 | retrieval and reasoning as context grows |

Acceptance target: no statistically meaningful regression versus the same
model on the GPU baseline. Record exact-match deltas, judge configuration where
used, invalid-output rate, truncations, and paired prompt-level disagreements.

## Coding and agentic suite

| Area | Proposed benchmark | Required reporting |
|---|---|---|
| Code generation | LiveCodeBench | pass@1, release/cutoff, language mix, execution sandbox |
| Repository repair | SWE-bench Verified | resolved %, scaffold/container version, attempts, cost/time budget |
| Harder repository repair | SWE-bench Pro | resolved %, same controls as above |
| Terminal agents | Terminal-Bench 2.0 | task score, agent harness, timeout, invalid actions |
| Function calling | BFCL v4 | overall, single-turn, multi-turn, agentic, hallucination, latency |
| Tool-use agents | tau2-bench | pass@k, domain, policy compliance, simulator version |
| MCP use | MCPMark and MCP-Atlas | task completion, call correctness, invalid-call rate |

Agent benchmarks measure the model *and the scaffold*. Publish the system
prompt, tool schemas, parser, retry policy, context compaction, allowed commands,
and maximum steps. Never compare scores from materially different scaffolds as
if they isolate the runtime.

## Private Genetec MCP qualification

Use a read-only replay or dedicated sandbox—never a production security system.
Build a versioned set of representative tool traces and measure:

- correct tool selection and argument-schema accuracy;
- invalid, hallucinated, or unnecessary calls;
- multi-step completion and recovery after tool errors;
- permission and policy compliance;
- final-answer factuality against replay ground truth;
- TTFT, total latency, tool latency, and tokens per completed task; and
- paired baseline/UMoX outcome agreement.

Redact sites, identities, credentials, hostnames, camera names, access-control
events, and customer information before publishing any artifact.

## Quality-preserving acceptance gates

A proposed optimization is accepted only when:

1. It beats a matched same-session control at p50 and does not create an
   unacceptable p95/p99 regression.
2. Numerical gates pass on held-out states, including decision-boundary cases.
3. Deterministic output parity passes for the parity corpus, or every difference
   is traced and justified under a predeclared tolerance.
4. Capability benchmark deltas remain within predeclared confidence bounds.
5. No hidden CPU fallback, graph spill, memory leak, or thermal collapse occurs.
6. Failures revert to a complete qualified path rather than mixing partial
   outputs from incompatible executions.

## Publication status

Current task-level UMoX scores for SWE-bench, LiveCodeBench, BFCL, Terminal-Bench,
MMLU-Pro, GPQA, and AIME are **not yet measured**. This document intentionally
does not convert upstream Qwen scores into UMoX claims.

Official benchmark references:

- [SWE-bench](https://www.swebench.com/)
- [Terminal-Bench](https://github.com/harbor-framework/terminal-bench)
- [BFCL](https://gorilla.cs.berkeley.edu/leaderboard)
- [LiveCodeBench](https://github.com/LiveCodeBench/LiveCodeBench)
- [LiveBench](https://livebench.ai/)
- [tau2-bench](https://github.com/sierra-research/tau2-bench)
- [GPQA](https://arxiv.org/abs/2311.12022)
- [MMLU-Pro](https://arxiv.org/abs/2406.01574)