# UMoX architecture

## Definition

**UMoX means Unified Mixture of Executors.** It is a quality-gated runtime
architecture in which CPU, GPU, and NPU are treated as hardware execution
experts for one resident model.

UMoX is inspired by a useful property of model-side mixture-of-experts systems:
specialization. The analogy stops there. The model's learned expert router is
unchanged; a separate runtime scheduler selects hardware for eligible subgraphs.

## Design goals

1. Keep one canonical model resident in unified memory when the platform allows.
2. Cache device-specific packed views only when their measured benefit repays
   memory, initialization, and synchronization costs.
3. Assign coarse, independent subgraphs instead of recomputing the full model.
4. Preserve all model work required by the selected inference mode: no expert
   dropping and no silent CPU fallback presented as NPU execution.
5. Treat numerical quality, stability, and recovery as scheduler constraints.
6. Fall back atomically to the qualified baseline when a route fails.

## Current Qwen3.6 slice

The transformer remains on the Adreno X2-90 GPU through llama.cpp/OpenCL. The
last hidden vector is passed to persistent ONNX Runtime QNN sessions that run
the 248,320-row language-model head on Hexagon v81.

The primary path uses two coalesced W8A16 shards. After the full-vocabulary
projection, the scheduler calculates the difference between the highest and
second-highest logit. If that margin is at or below `0.15`, it reruns only the
shards containing competitive candidates through W16A16 sessions. The refined
values replace their lower-precision counterparts before sampling.

The refinement rule is conservative and deterministic for a fixed build and
input. On the current 33-state calibration set it refined five states and
preserved all top-1 choices. During the 192-token live run it refined 59 of 193
decode calls, issuing 64 W16 shard invocations.

## Why the output projection was selected

The first production slice needed to be:

- large enough for the NPU to amortize graph dispatch;
- regular and stateless;
- independent from the GPU's KV-cache ownership;
- separable without forty per-layer device crossings; and
- measurable against exact full-vocabulary logits.

The language-model head meets those conditions. Earlier per-layer and MoE
experiments taught us that an operator running quickly in isolation does not
guarantee an end-to-end gain once handoffs, layout changes, launches, and
shared-memory contention are included.

## Precision as a scheduling decision

Pure W8A16 was the fastest measured path at 25.580 tok/s, but it changed one
top-1 result in a 33-state set. It was rejected. Full W16A16 passed with near
float-level agreement but costs more. The adaptive route uses W8 for common
states and spends W16 only when the logits indicate uncertainty.

This pattern generalizes beyond the head: fast paths should carry an error
budget and a confidence signal, with a higher-precision repair route for
states near a decision boundary.

## Data movement and coherency

All engines share physical memory bandwidth, but they do not automatically
share layouts, caches, ownership, or synchronization semantics. On the tested
Windows/ORT/QNN stack, session-scoped cached `QnnHtpShared` allocation worked
for immutable data but reused mutable host-written input could appear stale.
Current dynamic tensors therefore use explicit coherent staging. Small copies
are accepted when they are faster and more reliable than a fragile zero-copy
claim.

## Scheduler contract

A hardware route is eligible only if all of the following are true:

- its exact model, artifact, runtime, and driver hashes are known;
- required operators compile without spill or CPU fallback;
- the input shape and tensor layout match the qualified graph;
- measured p95 latency plus handoff cost beats the baseline;
- numerical and task-level quality gates pass;
- thermals and memory pressure are inside the qualified range; and
- failures can switch the whole affected operation to a known-good fallback.

Unknown latency is not zero. Unmeasured routes are ineligible.

## Near-term evolution

The current head offload is a deliberately coarse boundary. Candidate next
routes include low-rank residual repair, fewer mixed-precision sessions,
candidate-specific error bounds, multi-token verification, and coarse MoE
expert banks. CPU compute will be added only where it improves the measured
critical path rather than increasing contention.

The long-term objective is a persistent scheduler that observes latency,
precision, power, thermal state, and queue depth, then chooses among qualified
CPU/GPU/NPU plans without changing the model's expected behavior.

## Non-goals and present limitations

- UMoX does not promise that every processor must be busy every token.
- It does not infer token/s from advertised TOPS.
- It does not claim zero-copy for mutable GPU/NPU handoffs on this stack.
- It has not yet qualified complete transformer execution on the NPU.
- The Qwen release has one laptop, 33 calibration states, and one 192-token
  deterministic parity sequence; broad quality evaluation is still required.
- Cold startup creates ten QNN sessions and currently takes roughly two minutes
  to reach server readiness. The service is intended to remain resident.