QHexRT launch: 100% NPU, multimodal roadmap, remove open-source refs
Browse files
README.md
CHANGED
|
@@ -19,9 +19,13 @@ pipeline_tag: text-generation
|
|
| 19 |
# LFM2.5-230M — Hexagon v81 NPU (QHexRT)
|
| 20 |
|
| 21 |
[LiquidAI/LFM2.5-230M](https://huggingface.co/LiquidAI/LFM2.5-230M) running fully on the **Qualcomm Hexagon
|
| 22 |
-
v81 NPU** (Snapdragon 8 Elite Gen-2 / SM8850) via
|
| 23 |
-
|
| 24 |
-
Greedy output is identical to the source model
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
## Why the NPU — measured on SM8850 (vs llama.cpp CPU, same device)
|
| 27 |
|
|
@@ -34,7 +38,7 @@ Greedy output is identical to the source model (`"The capital of France is"` →
|
|
| 34 |
Batched **O(1) prefill** holds TTFT flat at **~36 ms regardless of prompt length**, so the NPU pulls further
|
| 35 |
ahead the longer the context — at far lower power than driving 8 CPU cores at max clock.
|
| 36 |
|
| 37 |
-
**Full
|
| 38 |
|
| 39 |
## Run
|
| 40 |
|
|
@@ -46,8 +50,8 @@ adb shell "cd /data/local/tmp/lfm230 && LD_LIBRARY_PATH=. \
|
|
| 46 |
```
|
| 47 |
|
| 48 |
Stage the QAIRT v81 runtime libs (`libQnnHtp.so`, `libQnnSystem.so`, `libQnnHtpV81Skel.so`/`Stub.so`) + the
|
| 49 |
-
`qhx_generate` tool into the same dir (from the QAIRT SDK
|
| 50 |
-
|
| 51 |
|
| 52 |
## `v81/`
|
| 53 |
`lfm2-5-230m.json` (manifest) · `lfm230_dec_512_w8.bin` (decode) · `lfm230_pf_512_w8.bin` (prefill) ·
|
|
|
|
| 19 |
# LFM2.5-230M — Hexagon v81 NPU (QHexRT)
|
| 20 |
|
| 21 |
[LiquidAI/LFM2.5-230M](https://huggingface.co/LiquidAI/LFM2.5-230M) running fully on the **Qualcomm Hexagon
|
| 22 |
+
v81 NPU** (Snapdragon 8 Elite Gen-2 / SM8850) via **QHexRT** — RunAnywhere's inference engine for
|
| 23 |
+
Qualcomm NPUs. **100% on the HTP.** No Python in the hot path. W8 weight-only, GQA-native decode,
|
| 24 |
+
batched prefill, on-NPU lm-head. Greedy output is identical to the source model
|
| 25 |
+
(`"The capital of France is"` → `" Paris."`).
|
| 26 |
+
|
| 27 |
+
QHexRT is the first engine built to run **LLM, VLM, STT, TTS, and embeddings** fully on Qualcomm
|
| 28 |
+
Hexagon NPUs. LFM 2.5 230M is the first model in the catalog.
|
| 29 |
|
| 30 |
## Why the NPU — measured on SM8850 (vs llama.cpp CPU, same device)
|
| 31 |
|
|
|
|
| 38 |
Batched **O(1) prefill** holds TTFT flat at **~36 ms regardless of prompt length**, so the NPU pulls further
|
| 39 |
ahead the longer the context — at far lower power than driving 8 CPU cores at max clock.
|
| 40 |
|
| 41 |
+
**Full launch write-up:** [runanywhere.ai/blog](https://www.runanywhere.ai/blog/qhexrt-npu-vs-cpu-lfm2.5-230m-qualcomm-hexagon) · [article draft in this repo](blog/qhexrt-npu-vs-cpu-lfm2.5-230m.md)
|
| 42 |
|
| 43 |
## Run
|
| 44 |
|
|
|
|
| 50 |
```
|
| 51 |
|
| 52 |
Stage the QAIRT v81 runtime libs (`libQnnHtp.so`, `libQnnSystem.so`, `libQnnHtpV81Skel.so`/`Stub.so`) + the
|
| 53 |
+
`qhx_generate` tool into the same dir (from the QAIRT SDK). Context binaries are **arch-pinned to v81**.
|
| 54 |
+
Contact [san@runanywhere.ai](mailto:san@runanywhere.ai) for QHexRT deployment access.
|
| 55 |
|
| 56 |
## `v81/`
|
| 57 |
`lfm2-5-230m.json` (manifest) · `lfm230_dec_512_w8.bin` (decode) · `lfm230_pf_512_w8.bin` (prefill) ·
|