QHexRT launch article: full-stack NPU inference, not open source

blog/qhexrt-npu-vs-cpu-lfm2.5-230m.md

@@ -1,42 +1,41 @@
-# QHexRT: Inference Engine for Qualcomm NPUs
+# QHexRT Is Live: Full-Stack NPU Inference for Qualcomm Hexagon
 
-We shipped MetalRT for Apple Silicon. Now we're bringing the same approach to Qualcomm with **QHexRT**: a custom inference runtime built for **every Qualcomm Hexagon NPU**.
+We shipped MetalRT for Apple Silicon — the first engine to run LLM, speech, vision, and speech-to-speech in one runtime, entirely on the GPU. Today we're launching **QHexRT** for Qualcomm: the same bet on the NPU.
 
-**QHexRT runs on Qualcomm Hexagon NPUs across the fleet.** More architectures ship continuously. Same-day model support, full NPU performance, one runtime.
+**QHexRT runs inference 100% on the Hexagon NPU.** No Python in the hot path. No CPU fallback during inference. We are building the widest model catalog on any Qualcomm NPU stack, with same-day support for the models the community ships.
 
-We're building QHexRT with one goal: **same-day NPU support for the best models the community ships.** When a model drops, you should be able to run it on the NPU within hours, not months later.
+No one has shipped a single runtime that covers **LLM, VLM, STT, TTS, and embeddings** fully on Qualcomm NPUs. MetalRT did it for Apple Silicon. QHexRT does it for Hexagon.
 
-**LiquidAI released [LFM 2.5 230M](https://huggingface.co/LiquidAI/LFM2.5-230M) on June 25, 2026.** QHexRT supports it on day one. The NPU bundle is here: [runanywhere/lfm2_5_230m_HNPU](https://huggingface.co/runanywhere/lfm2_5_230m_HNPU).
+## First model: LFM 2.5 230M
 
-This post is the first public benchmark. We ran LFM 2.5 230M on **Hexagon v81** (Snapdragon 8 Elite Gen-2, SM8850) on both the NPU and the CPU of the same chip, same die, same phone. Both stacks produce identical output. The only variable is speed.
+**LiquidAI released [LFM 2.5 230M](https://huggingface.co/LiquidAI/LFM2.5-230M) on June 25, 2026.** QHexRT supports it on day one — our first catalog entry. The NPU bundle is here: [runanywhere/lfm2_5_230m_HNPU](https://huggingface.co/runanywhere/lfm2_5_230m_HNPU).
 
-## The headline numbers
+Every tensor in the inference path stays on the HTP: decode graph, prefill graph, lm-head, embeddings. Greedy output matches the source model (`"The capital of France is"` → `" Paris."`).
 
-- **Prefill: 12,540 tok/s** on the NPU vs 871 on the CPU (Q8_0). **14.4x faster.**
-- **Time-to-first-token: 36ms flat** for any prompt up to 512 tokens. The CPU takes 588ms at 512 tokens.
-- **Decode: the CPU wins** at 250 tok/s vs 172 on the NPU. At 230M params, decode is memory-bandwidth-bound, and the Oryon CPU pulls more usable bandwidth from DDR on this weight set.
-- **End-to-end: the NPU wins** once the prompt exceeds ~1.5x the generation length.
+## All modalities, one NPU runtime
 
-Both stacks emit the same greedy token chain. QHexRT's W8 graph matched the HuggingFace fp32 oracle at logits cosine 1.000000. This is a speed comparison at equal quality.
+| Modality | Status |
+| --- | --- |
+| **LLM** | Live — LFM 2.5 230M on Hexagon v81 |
+| **VLM** | In development |
+| **STT** | In development |
+| **TTS** | In development |
+| **Embeddings** | In development |
 
-## Setup
+One engine, one deployment model, every modality on the NPU — the same architecture MetalRT proved on Apple Silicon, now on Qualcomm.
 
-| Item | Value |
-| --- | --- |
-| **Device** | Qualcomm SM8850 (Snapdragon 8 Elite Gen-2 class) |
-| **NPU** | Hexagon v81 (HTP) |
-| **CPU** | 8-core Oryon (up to ~4.4 GHz) |
-| **Model** | [LiquidAI/LFM 2.5 230M](https://huggingface.co/LiquidAI/LFM2.5-230M) (229.69M params, hybrid decoder) |
-| **NPU bundle** | [runanywhere/lfm2_5_230m_HNPU](https://huggingface.co/runanywhere/lfm2_5_230m_HNPU) (v81 context binaries, W8) |
-| **NPU stack** | QHexRT, W8 weight-only (int8 weights, fp16 activations) |
-| **CPU stack** | llama.cpp b1-beac530, Q8_0 + Q4_K_M |
-| **Fairness** | HTP clock pinned to TURBO; CPU governor pinned to performance, all 8 cores |
+## The headline numbers (LFM 2.5 230M · Hexagon v81)
 
-LFM 2.5 230M is a hybrid decoder: 14 layers (6 GQA-attention + 8 short-convolution), hidden size 1024, 16 query heads / 8 KV heads, 65K vocab, tied lm-head. The small query-head count is what makes GQA-native NPU decode possible here.
+Benchmarked on **Hexagon v81** (Snapdragon 8 Elite Gen-2, SM8850). We also ran llama.cpp on the CPU of the same chip, same die, same phone. Both stacks produce identical output.
 
-## Run it
+- **Prefill: 12,540 tok/s** on the NPU vs 871 on the CPU (Q8_0). **14.4x faster.**
+- **Time-to-first-token: 36ms flat** for any prompt up to 512 tokens. The CPU takes 588ms at 512 tokens.
+- **Decode: the CPU wins** at 250 tok/s vs 172 on the NPU. At 230M params, decode is memory-bandwidth-bound.
+- **End-to-end: the NPU wins** once the prompt exceeds ~1.5x the generation length.
+
+QHexRT's W8 graph matched the HuggingFace fp32 oracle at logits cosine 1.000000.
 
-Download the v81 NPU bundle from [runanywhere/lfm2_5_230m_HNPU](https://huggingface.co/runanywhere/lfm2_5_230m_HNPU). It includes the QNN context binaries (decode, prefill, lm-head), embeddings, tokenizer, and manifest. No Python in the hot path.
+## Run it
 
 ```bash
 hf download runanywhere/lfm2_5_230m_HNPU --local-dir lfm2_5_230m_HNPU
@@ -45,86 +44,49 @@ adb shell "cd /data/local/tmp/lfm230 && LD_LIBRARY_PATH=. \
   ./qhx_generate lfm2-5-230m.json libQnnHtp.so libQnnSystem.so . 64 'The capital of France is'"
 ```
 
-Stage the QAIRT v81 runtime libs (`libQnnHtp.so`, `libQnnSystem.so`, `libQnnHtpV81Skel.so`/`Stub.so`) and the `qhx_generate` tool into the same directory. See the [QHexRT deploy docs](https://github.com/RunanywhereAI/QHexRT) for details. Context binaries are pinned to Hexagon v81.
-
-Greedy output matches the source model: `"The capital of France is"` → `" Paris."`
+Stage the QAIRT v81 runtime libs and the `qhx_generate` tool into the same directory. Context binaries are pinned to Hexagon v81.
 
 ## Throughput
 
-| Engine | Prefill (tok/s) | Decode (tok/s) | Decode (ms/tok) | Peak RAM (MB) |
-| --- | ---: | ---: | ---: | ---: |
-| **QHexRT NPU** (v81, W8) | **12,540** | 172 | 5.8 | 445 |
-| **llama.cpp CPU** Q8_0 | 871 | **250** | 4.0 | 299 |
-| **llama.cpp CPU** Q4_K_M | 680 | **264** | 3.83 | 209 |
-
-Prefill is where the NPU wins by the widest margin. QHexRT runs it as one batched forward over a padded 512-token window, so the cost stays constant regardless of prompt length.
+| Engine | Prefill (tok/s) | Decode (tok/s) | Peak RAM (MB) |
+| --- | ---: | ---: | ---: |
+| **QHexRT NPU** (v81, W8) | **12,540** | 172 | 445 |
+| **llama.cpp CPU** Q8_0 | 871 | **250** | 299 |
+| **llama.cpp CPU** Q4_K_M | 680 | **264** | 209 |
 
-![Prefill throughput on SM8850 — QHexRT NPU vs llama.cpp CPU](https://huggingface.co/runanywhere/lfm2_5_230m_HNPU/resolve/main/assets/prefill_throughput.png)
+![Prefill throughput — LFM 2.5 230M on SM8850](https://huggingface.co/runanywhere/lfm2_5_230m_HNPU/resolve/main/assets/prefill_throughput.png)
 
-**Figure 1 — Prefill throughput (log scale).** QHexRT on Hexagon v81 hits **12,540 tok/s** prefill. llama.cpp on the same die's Oryon CPU reaches 871 tok/s (Q8_0) and 680 tok/s (Q4_K_M). That is **14.4x** and **18.4x** faster prefill on the NPU.
-
-Decode tells the opposite story on this 230M model. The CPU wins by 1.45–1.53x. Single-token decode reads the full weight set once per token and does minimal compute. At 230M params it is DDR-bandwidth-bound. The Oryon CPU pulls ~58 GB/s effective bandwidth on this weight set vs ~40 GB/s on the HTP. Larger models shift toward compute-bound decode, where the NPU's HMX units have the advantage.
+**Figure 1 — Prefill throughput (log scale).** QHexRT on Hexagon v81 hits **12,540 tok/s** prefill on LFM 2.5 230M. **14.4x** faster than llama.cpp CPU Q8_0 on the same die.
 
 ## Time to first token
 
-| Prompt (tokens) | NPU (ms) | CPU Q8_0 (ms) | CPU Q4_K_M (ms) |
-| ---: | ---: | ---: | ---: |
-| 16 | **36** | 17 | 26 |
-| 128 | **36** | 132 | 188 |
-| 512 | **36** | 588 | 753 |
-
-![Time to first token vs prompt length — flat NPU line vs rising CPU](https://huggingface.co/runanywhere/lfm2_5_230m_HNPU/resolve/main/assets/ttft_vs_prompt.png)
-
-**Figure 2 — Time to first token vs prompt length.** The NPU holds a flat **~36 ms** TTFT for any prompt up to 512 tokens because prefill runs as one batched forward pass. The CPU must stream every prompt token through the weights, so TTFT grows linearly. At 512 tokens the NPU delivers its first token **16x sooner** than Q8_0 (36 ms vs 588 ms). For prompts under ~30 tokens the CPU is still faster; past that crossover the NPU wins by a widening margin.
-
-## End-to-end latency
-
-End-to-end combines a one-time TTFT with G decode steps:
-
-| Prompt (tokens) | NPU (ms) | CPU Q8_0 (ms) | CPU Q4_K_M (ms) |
-| ---: | ---: | ---: | ---: |
-| 16 | 773 | **559** | **513** |
-| 128 | 773 | **674** | 675 |
-| 512 | **773** | 1130 | 1239 |
-
-At G=128, the NPU holds constant at 773ms regardless of prompt length. The CPU starts faster on short prompts but its rising prefill cost overtakes the NPU near a ~200-token prompt. A real run (15-token prompt + 128 generated tokens) measured 777ms. The formula predicted 772ms. 1% error.
+| Prompt (tokens) | NPU (ms) | CPU Q8_0 (ms) |
+| ---: | ---: | ---: |
+| 16 | **36** | 17 |
+| 128 | **36** | 132 |
+| 512 | **36** | 588 |
 
-The NPU wins end-to-end once prompt length exceeds roughly **1.5x the generation length** (e.g. for G=128, once the prompt passes ~200 tokens).
+![TTFT vs prompt length — LFM 2.5 230M](https://huggingface.co/runanywhere/lfm2_5_230m_HNPU/resolve/main/assets/ttft_vs_prompt.png)
 
-## When each engine wins
-
-**Use QHexRT (NPU)** for prefill-heavy or latency-sensitive workloads: RAG, summarization, long-context requests, classification, extraction, scoring. The flat TTFT keeps the first token fast regardless of prompt length. The NPU also frees the CPU for the rest of the app and uses less power.
-
-**Use the CPU** for short-prompt, long-generation chat on tiny models (under ~500M params), where raw decode tok/s dominates and prompts stay well under the crossover.
-
-Peak RAM: QHexRT NPU 445 MB, llama.cpp Q8_0 299 MB, Q4_K_M 209 MB. The NPU keeps int8 weights, the QNN context, and persistent graph I/O tensors resident.
+**Figure 2 — Time to first token (LFM 2.5 230M).** Flat **~36 ms** TTFT on the NPU for any prompt up to 512 tokens.
 
 ## The model catalog
 
-[LFM 2.5 230M on Hexagon v81](https://huggingface.co/runanywhere/lfm2_5_230m_HNPU) is the first model in the QHexRT catalog. QHexRT targets all Qualcomm Hexagon NPUs. We're adding models as fast as the community ships them.
+[LFM 2.5 230M on Hexagon v81](https://huggingface.co/runanywhere/lfm2_5_230m_HNPU) is the first entry. We're adding models as fast as the community ships them.
 
-This is one small model on one SoC. The decode result is size-dependent. Larger models are more compute-bound, and we expect the NPU to win decode too as HMX utilization rises and the bandwidth wall recedes.
-
-Next on the roadmap:
-
-- **More models.** Qwen3, Gemma 3, Phi-4-mini, and other sub-4B models that fit the NPU's memory budget.
-- **W4 quantization.** A working W4 path would roughly halve the per-token bytes, projecting ~300-340 tok/s decode.
-- **Power metering.** The CPU numbers come from 8 Oryon cores pinned at max clock. The NPU does the same work at a fraction of the power.
+Next: **VLM, STT, TTS, embeddings** on Hexagon, more LLM models, W4 quantization, and power metering.
 
 ## Summary
 
-- **Day-one NPU support** for LFM 2.5 230M (released June 25, 2026)
-- **NPU bundle:** [runanywhere/lfm2_5_230m_HNPU](https://huggingface.co/runanywhere/lfm2_5_230m_HNPU)
-- **12,540 tok/s** prefill (14.4x faster than CPU Q8_0)
-- **36ms** flat TTFT for any prompt up to 512 tokens
-- **172 tok/s** decode (CPU wins on this 230M model)
-- NPU wins end-to-end once prompt length exceeds ~1.5x generation length
-- Identical greedy output, W8 logits cosine 1.000000 vs fp32 oracle
-
-LFM 2.5 230M is the first model in the QHexRT catalog. More models, more SoCs, more benchmarks are coming.
+- **QHexRT is live** — full-stack NPU inference for Qualcomm Hexagon
+- **100% on NPU** — no Python, no CPU fallback during inference
+- **First model:** LFM 2.5 230M ([runanywhere/lfm2_5_230m_HNPU](https://huggingface.co/runanywhere/lfm2_5_230m_HNPU))
+- **12,540 tok/s** prefill · **36ms** flat TTFT
+- **All modalities** — LLM live; VLM, STT, TTS, embeddings in development
+- **Widest model catalog** for Qualcomm NPUs, expanding continuously
 
-QHexRT is open source: [github.com/RunanywhereAI/QHexRT](https://github.com/RunanywhereAI/QHexRT)
+Full write-up: [runanywhere.ai/blog](https://www.runanywhere.ai/blog/qhexrt-npu-vs-cpu-lfm2.5-230m-qualcomm-hexagon)
 
 ---
 
-_Benchmarked on Qualcomm SM8850 / Hexagon v81. Model: LiquidAI/LFM 2.5 230M. NPU: QHexRT W8 weight-only, QAIRT 2.47.0.260601. CPU: llama.cpp b1-beac530, Q8_0 + Q4_K_M, 8 threads. Both engines produce identical greedy output._
+_Benchmarked on Qualcomm SM8850 / Hexagon v81. Model: LiquidAI/LFM 2.5 230M. NPU: QHexRT W8 weight-only, QAIRT 2.47.0.260601._