docs/FABLE_HANDOFF_ROUND3.md

UMoX Round 3 handoff for Fable

Tone and role

Fable, thank you for helping as an independent second opinion. UMoX is still a prototype and needs substantial polishing. Please be candid but constructive: try to falsify the claims, preserve the rejected results, and propose the next small experiment that could prove us wrong. Do not edit the active runtime, download large models, submit cloud jobs, expose credentials, or change public claims. Return a written audit and a prioritized experiment proposal.

Common goal

Maximize accurate, stable, sustained batch-1 local inference on a Snapdragon X2 Elite Extreme laptop without “dumbing down” the model. One checkpoint should remain resident while CPU, Adreno GPU, and Hexagon NPU receive different useful subgraphs. No engine should recompute the whole model merely to appear utilized.

UMoX means Unified Mixture of Executors: hardware engines act as execution experts selected by a runtime scheduler. This is separate from and must not alter the learned MoE router inside Qwen/Ornith.

Hardware and primary checkpoint

• Snapdragon X2 Elite Extreme, 18-core Oryon CPU.
• Adreno X2-90 GPU.
• 80-TOPS-class Hexagon v81 NPU.
• Unified LPDDR5x memory, advertised up to 228 GB/s.
• Windows 11 ARM64.
• Current production target: Qwen3.6-35B-A3B Q5_K_M GGUF.
• Model architecture: about 35B total / 3B active parameters, hidden size 2048, vocabulary 248,320, 40 layers, 256 routed experts, top 8 plus shared expert.

Peak TOPS is not expected token throughput. Batch-1 autoregressive decode is limited by memory movement, serial dependencies, graph launch and fence cost, supported operator shapes, quantization boundaries, and the fraction of work that is sufficiently wide to use NPU arithmetic.

Journey: observations that shaped UMoX

1. Initial model-level orientation

Early measurements established useful but not directly comparable baselines:

Model/path	Historical result	Interpretation
Qwen3-4B-Instruct-2507 QAIRT	26.91 tok/s NPU	real NPU use, but TOPS alone did not translate to expected decode rate
Qwen3.5-4B Q4_0	18.09 tok/s NPU	architecture/runtime compatibility matters more than parameter count alone
Qwen3.6-35B-A3B Q4_0	16.60 tok/s GPU	early orientation before the final Q5_K_M path
Ornith 35B Q4_K_M	about 20.6 tok/s GPU	useful MoE comparison
Ornith 9B Q4_K_M	18.23 tok/s GPU	faster and smarter quantization choice than Q6_K on this machine
Ornith 9B Q6_K	14.65 tok/s GPU	extra weight precision reduced throughput materially

Some early tests occurred on battery or under changing power modes. Treat them as historical orientation, not a controlled leaderboard.

2. Operator success did not imply end-to-end success

QAIRT successfully compiled fixed real-weight layer and expert graphs entirely to HTP. An Ornith real layer W4A16 measured 1.0759 ms locally and 0.999 ms on a hosted device; W4A8 was only about 4.9% faster locally. Grouped GQA reduced a 4K-attention component from 27.206 ms to 1.672 ms by matching the workload shape to supported dense operations.

However, dynamic selection of quantized expert weights failed compilation. Per-layer/per-expert contexts and repeated hidden-state crossings threatened to erase the component advantage. A fast graph is only a candidate until its layout, dispatch, fences, and full decoder critical path are measured.

3. Generic graph-wide INT4 was rejected for quality

Against float checkpoint outputs on real prompt states:

• W4A16 whole-layer output: cosine about 0.598, NRMSE about 1.472.
• W4A8 with 8 calibration states: cosine about 0.512, NRMSE about 1.416.
• W4A8 with 128 states from diverse prompts: cosine about 0.532, NRMSE about 1.396.

More calibration did not repair an unsuitable precision/layout choice. Float ONNX matched PyTorch and locally lowered QDQ matched the original QDQ, locating the major error in generic INT4 quantization rather than export or lowering.

4. Mixed precision and coarse boundaries worked

An accepted Ornith split kept sensitive DeltaNet/norm/residual work in FP16 and used W4A16 for selected MoE work. It reached cosine 0.997818 and NRMSE 0.0740. Later quality-aware expert caching and layer replay established these principles:

• exact selected experts, never dropped;
• immutable resident NPU banks only for qualified experts;
• CPU FP32 cold/shared paths when they improved quality or recovery;
• disjoint output buffers and explicit FP32 merge;
• route eligibility based on measured cost, not the desire to use every engine;
• atomic fail-closed fallback.

These component/layer replays were important architectural evidence, but they were not end-to-end token/s claims.

5. Unified memory helped residency, not automatic coherency

Native ARM64 QNN testing showed that session-scoped cached QnnHtpShared allocation was effective for immutable data. Reusing a host-mutated cached shared buffer could produce stale NPU input even after the exposed synchronize call. Explicit coherent CPU staging for dynamic 4 KiB tensors was effectively latency-neutral in the accepted expert bundle and was safer.

Raw Vulkan external-memory import also failed because compatible host and storage-buffer memory-type masks did not intersect. The practical lesson is that “unified memory” does not remove cache maintenance, layout conversion, ownership, or fence requirements. Small explicit copies can be the polished solution when their cost is measured and bounded.

6. The output projection became the first production slice

We chose Qwen3.6's language-model head because it is large, regular, stateless, full-width, separable from KV ownership, and crossed once per token rather than once per transformer layer.

The transformer remains on Adreno. The final FP32 hidden vector feeds a complete 248,320-row NPU projection. Ten persistent QNN sessions are loaded:

• two full-coverage W8A16 primary shards; and
• eight W16A16 refinement shards.

The primary W8 result is inspected after every token. If the top-logit margin is at or below 0.15, only shards containing competitive candidates are rerun at W16 and patched into the logits before sampling. This is adaptive precision, not speculative text correction and not full-head recomputation.

Final qualified Qwen3.6 result

Power state: AC connected, battery 95–97%, ASUS Performance mode. Batch 1.

End-to-end decode

Path	tok/s	Status
GPU control A	21.391753	reference
GPU control B	23.591248	faster trailing reference
Two-control mean	22.491501	reference mean
UMoX adaptive W8 + W16	24.257476	accepted release candidate
Static W16-lower/W8-upper	23.810490	accepted component/profile
Full W8	25.580063	rejected for one top-1 flip

Conservative gain versus the faster trailing control: +2.824%. Gain versus the two-control mean: +7.852%. The material control drift is disclosed; the smaller figure is the public headline.

Adaptive component quality

• 33 calibration states; 33/33 exact global top-1.
• Mean top-5 overlap 1.0; mean top-10 overlap 0.9878788.
• Cosine 0.999956645; NRMSE 0.00933829.
• Mean KL divergence 0.000194641.
• Median component latency 6.8199 ms; p95 9.73378 ms.
• Five of 33 offline states triggered refinement.
• During live decode, 59/193 calls refined, with 64 W16 shard invocations.
• Live NPU-head median 4.551 ms.

Quality controls and alternatives

Full W16 achieved 33/33 top-1, cosine 0.999999966, NRMSE 0.000262638, and mean KL 1.058e-7. It is the maximum-precision component reference; its full live benchmark remains pending.

Full W8 was faster but produced 32/33 top-1. Widening output QDQ ranges to ±32 eliminated clipping, yet weight quantization still flipped a 0.01696-logit near-tie. It remains explicitly rejected, which motivated adaptive refinement.

A 192-token greedy A/B generation produced exact byte-for-byte output with a matching SHA-256. An OpenAI-compatible server smoke test passed health, model listing, and chat at 24.3918 tok/s. This behavioral evidence is narrow, not a substitute for task benchmarks.

What is genuinely established

1. Useful NPU work can be inserted into a live Qwen3.6 GPU decoder and beat a matched same-session control on this machine.
2. The runtime does not recompute the whole transformer on multiple devices.
3. Precision can be scheduled dynamically around a measurable decision boundary.
4. A fastest path may—and did—get rejected when it changed top-1 behavior.
5. Keeping the service resident amortizes expensive QNN context initialization.
6. Coarse partitions can outperform ambitious per-layer shuttling.

What remains unproven

• Generalization beyond one X2 laptop and the qualified driver/runtime stack.
• Broad task-level parity for reasoning, coding, instructions, and agents.
• Energy efficiency, 10/60-minute thermal stability, and long-context scaling.
• A useful three-way CPU+GPU+NPU critical path in the production Qwen decoder.
• A robust mutable zero-copy GPU/NPU handoff.
• Complete NPU transformer or MoE execution that beats the current GPU route.
• A calibrated proof that margin 0.15 covers unseen decision boundaries.
• Fast cold startup; ten QNN sessions take about 42.6 seconds to initialize and total server readiness is roughly 2 minutes 15 seconds.

Public benchmark plan

UMoX must rerun the same checkpoint under the same scaffold on GPU baseline and UMoX. Planned suites:

• reasoning/instruction: LiveBench, MMLU-Pro, GPQA Diamond, AIME, IFEval;
• coding: LiveCodeBench, SWE-bench Verified, SWE-bench Pro;
• agentic/tool use: Terminal-Bench 2.0, BFCL v4, tau2-bench, MCPMark, MCP-Atlas;
• private Genetec MCP: redacted read-only replay for schema accuracy, invalid calls, multi-step completion, policy compliance, recovery, and latency;
• systems: TTFT, prefill/decode p50/p95/p99, memory, joules/token, cold start, long-context scaling, concurrency, soak, and sleep/resume.

The upstream Qwen card reports strong task scores, but none are UMoX results until rerun. UMoX should preserve capability; it cannot inherit a benchmark claim merely by using the same weights.

Requested independent audit

Please focus on these falsifiable questions:

1. Is raw top-1/top-2 margin a sufficient uncertainty signal? Derive a better candidate-specific bound using per-shard quantization error, logit scale, top-k separation, or calibrated conformal risk.
2. Can W16 refinement be replaced or supplemented by a small low-rank residual correction without masking rare quantization failures?
3. Can two W8 plus eight W16 sessions be reduced through a mixed-precision QNN context, linked graph, or different shard geometry while preserving gates?
4. Which NPU work can overlap safely with Adreno without both saturating unified bandwidth or creating a serial fence on the token critical path?
5. Is multi-token/speculative verification a better way to widen NPU work? Model acceptance rate, verifier batch width, extra memory traffic, and rollback cost.
6. Can expert banks or DeltaNet components form a coarse second boundary that crosses devices only once per multi-layer block?
7. What minimum calibration corpus and statistical bound justify the refinement threshold on unseen prompts?
8. Which next experiment offers the highest information gain per compile hour and per gigabyte of storage?

For each proposal, provide:

• a mathematical critical-path model;
• expected p50 and p95 impact, not only optimistic throughput;
• memory traffic and resident-storage estimate;
• numerical and task-quality risks;
• an exact baseline and experiment order;
• acceptance/rejection thresholds declared in advance; and
• the measurement that would falsify the idea.

Recommended next experiment

Our current first choice is a per-shard error-envelope refinement gate:

1. collect a larger held-out hidden-state corpus spanning coding, reasoning, tool calling, long context, and adversarial near-ties;
2. measure W8-versus-W16 error distributions per shard and candidate rank;
3. derive an upper error bound for every contender within a dynamic window;
4. refine only where error intervals overlap the apparent winner;
5. compare against the fixed 0.15 margin on top-1 misses, refinement rate, component p95, live tok/s, and task parity.

Provisional acceptance: zero additional top-1 misses on the held-out corpus, no worse task-level parity, fewer W16 invocations or lower p95 than the fixed margin, and an end-to-end gain that survives an A/B/A control.

Please challenge this recommendation rather than accepting it by default. The goal is not to defend UMoX; it is to make the next version more correct.

Closing note

We appreciate the thought and skepticism you bring to this project. UMoX has moved from a hopeful hardware idea to a small, measured end-to-end improvement, but it still has a long way to go. A careful criticism, failed reproduction, or simpler design may be exactly what helps it mature. Thank you for helping us build it openly and responsibly.