ARCHITECTURE.md

UMoX architecture

Definition

UMoX means Unified Mixture of Executors. It is a quality-gated runtime architecture in which CPU, GPU, and NPU are treated as hardware execution experts for one resident model.

UMoX is inspired by a useful property of model-side mixture-of-experts systems: specialization. The analogy stops there. The model's learned expert router is unchanged; a separate runtime scheduler selects hardware for eligible subgraphs.

Design goals

1. Keep one canonical model resident in unified memory when the platform allows.
2. Cache device-specific packed views only when their measured benefit repays memory, initialization, and synchronization costs.
3. Assign coarse, independent subgraphs instead of recomputing the full model.
4. Preserve all model work required by the selected inference mode: no expert dropping and no silent CPU fallback presented as NPU execution.
5. Treat numerical quality, stability, and recovery as scheduler constraints.
6. Fall back atomically to the qualified baseline when a route fails.

Current Qwen3.6 slice

The transformer remains on the Adreno X2-90 GPU through llama.cpp/OpenCL. The last hidden vector is passed to persistent ONNX Runtime QNN sessions that run the 248,320-row language-model head on Hexagon v81.

The primary path uses two coalesced W8A16 shards. After the full-vocabulary projection, the scheduler calculates the difference between the highest and second-highest logit. If that margin is at or below 0.15, it reruns only the shards containing competitive candidates through W16A16 sessions. The refined values replace their lower-precision counterparts before sampling.

The refinement rule is conservative and deterministic for a fixed build and input. On the current 33-state calibration set it refined five states and preserved all top-1 choices. During the 192-token live run it refined 59 of 193 decode calls, issuing 64 W16 shard invocations.

Why the output projection was selected

The first production slice needed to be:

• large enough for the NPU to amortize graph dispatch;
• regular and stateless;
• independent from the GPU's KV-cache ownership;
• separable without forty per-layer device crossings; and
• measurable against exact full-vocabulary logits.

The language-model head meets those conditions. Earlier per-layer and MoE experiments taught us that an operator running quickly in isolation does not guarantee an end-to-end gain once handoffs, layout changes, launches, and shared-memory contention are included.

Precision as a scheduling decision

Pure W8A16 was the fastest measured path at 25.580 tok/s, but it changed one top-1 result in a 33-state set. It was rejected. Full W16A16 passed with near float-level agreement but costs more. The adaptive route uses W8 for common states and spends W16 only when the logits indicate uncertainty.

This pattern generalizes beyond the head: fast paths should carry an error budget and a confidence signal, with a higher-precision repair route for states near a decision boundary.

Data movement and coherency

All engines share physical memory bandwidth, but they do not automatically share layouts, caches, ownership, or synchronization semantics. On the tested Windows/ORT/QNN stack, session-scoped cached QnnHtpShared allocation worked for immutable data but reused mutable host-written input could appear stale. Current dynamic tensors therefore use explicit coherent staging. Small copies are accepted when they are faster and more reliable than a fragile zero-copy claim.

Scheduler contract

A hardware route is eligible only if all of the following are true:

• its exact model, artifact, runtime, and driver hashes are known;
• required operators compile without spill or CPU fallback;
• the input shape and tensor layout match the qualified graph;
• measured p95 latency plus handoff cost beats the baseline;
• numerical and task-level quality gates pass;
• thermals and memory pressure are inside the qualified range; and
• failures can switch the whole affected operation to a known-good fallback.

Unknown latency is not zero. Unmeasured routes are ineligible.

Near-term evolution

The current head offload is a deliberately coarse boundary. Candidate next routes include low-rank residual repair, fewer mixed-precision sessions, candidate-specific error bounds, multi-token verification, and coarse MoE expert banks. CPU compute will be added only where it improves the measured critical path rather than increasing contention.

The long-term objective is a persistent scheduler that observes latency, precision, power, thermal state, and queue depth, then chooses among qualified CPU/GPU/NPU plans without changing the model's expected behavior.

Non-goals and present limitations

• UMoX does not promise that every processor must be busy every token.
• It does not infer token/s from advertised TOPS.
• It does not claim zero-copy for mutable GPU/NPU handoffs on this stack.
• It has not yet qualified complete transformer execution on the NPU.
• The Qwen release has one laptop, 33 calibration states, and one 192-token deterministic parity sequence; broad quality evaluation is still required.
• Cold startup creates ten QNN sessions and currently takes roughly two minutes to reach server readiness. The service is intended to remain resident.