| --- |
| license: apache-2.0 |
| tags: |
| - qhexrt |
| - internvl |
| - vlm |
| - image-text-to-text |
| - hexagon |
| - npu |
| - qnn |
| base_model: OpenGVLab/InternVL3_5-1B |
| pipeline_tag: image-text-to-text |
| language: |
| - en |
| --- |
| |
| # InternVL3.5-1B — QHexRT NPU bundle (Hexagon v79) |
|
|
| Precompiled **InternVL3.5-1B** vision-language model for the **QHexRT** runtime on Qualcomm |
| Hexagon **v79** (Snapdragon 8 Elite / SM8750, e.g. Galaxy S25). Image → caption runs **entirely on |
| the NPU**: the InternViT-300M vision encoder and the full Qwen3-0.6B language model are |
| `qnn_context_binary` graphs; the host does only light glue (image resize, patch-embed, pixel-shuffle |
| projector, tokenize/splice, sampling). Device-validated end-to-end on diverse photos. |
|
|
| The model is driven by QHexRT's declarative **manifest plan-interpreter** (`Session::generate` via the |
| `internvl_generate` host-op) — the same engine that runs the other `*_HNPU` bundles. |
|
|
| ## Contents (`v79/`) |
| | file | what | size | |
| |------|------|------| |
| | `internvl3_5-1b-512.json` | QHexRT manifest (vlm family, `internvl_generate` plan) | — | |
| | `ivl_enc2.bin` | InternViT-300M vision encoder — 24-layer, **fp16**, 2D `[seq,dim]` I/O | 621 MB | |
| | `ivl_pf_512.bin` | Qwen3-0.6B **batched prefill** graph, AR=512 (padded) — **arbitrary prompts** up to ~256 text tokens (**fp16**) | 890 MB | |
| | `ivl_dec_w4.bin` | Qwen3-0.6B **decode** step (1 token), **W4A16** | 225 MB | |
| | `ivl_lmh_q.bin` | untied **lm-head**, **W8A16** (int8) → logits over 151936 vocab | 158 MB | |
| | `ivl_embed_f16.bin` | token embedding table `[151936,1024]` f16 (mmap, host lookup) | 311 MB | |
| | `tokenizer.json` | Qwen3 tokenizer (vocab 151936) | 11 MB | |
| | `vlm/` | host vision preproc: Conv2d patch-embed, pos-embed, layer-norm + pixel-shuffle MLP projector, prompt template | 27 MB | |
|
|
| Total ≈ **2.25 GB** on disk; ≈ **2.0 GB** peak device RSS at runtime (weights are ion-shared host↔DSP, |
| not duplicated). |
|
|
| ## Run (QHexRT CLI) |
| ```bash |
| hf download runanywhere/internvl3_5_1b_HNPU --local-dir internvl3_5_1b_HNPU |
| # qhx_generate comes from a QHexRT build; QNN libs from the QAIRT SDK (lib/aarch64-android) + v79 HTP skel. |
| adb push internvl3_5_1b_HNPU/v79 /data/local/tmp/wq/internvl |
| adb push my_photo.jpg /data/local/tmp/wq/internvl/ |
| |
| adb shell "cd /data/local/tmp/wq && LD_LIBRARY_PATH=. ADSP_LIBRARY_PATH=. \ |
| ./qhx_generate internvl/internvl3_5-1b-512.json libQnnHtp.so libQnnSystem.so internvl \ |
| 60 'What animals are in this picture and what are they doing?' internvl/my_photo.jpg" |
| # -> text: "In the image, there are two cats lying on a pink couch ..." |
| ``` |
| - **Image**: any aspect ratio / format (JPEG/PNG/…); center-cropped + resized to 448×448 host-side |
| (256 image tokens / 1 tile). |
| - **Arbitrary prompts**: the padded `AR=512` prefill accepts any prompt up to ~256 text tokens (image is |
| fixed at 256 tokens). Ask real questions, not just a fixed caption string. |
|
|
| ## Performance (measured, v79 / S25) |
| | stage | time | |
| |-------|------| |
| | vision encode (one-time) | **181 ms** | |
| | prefill / TTFT | **101 ms** | |
| | decode | **~28 ms/token** on the NPU graphs | |
|
|
| End-to-end wall time includes single-threaded host preproc (patch-embed + projector, ~1.5 s one-time); |
| the NPU compute itself is the table above. |
|
|
| ## Notes |
| - Arch: **v79 only** — context binaries are dsp-arch-pinned. Other arches must be re-exported. |
| - No custom op-package needed — all graphs are pure-native HTP (HVX/HMX). |
| - **Precision**: vision fp16, prefill fp16, decode **W4A16**, lm-head **W8A16**. (Prefill/vision can be |
| further quantized to shrink the footprint; this bundle ships the validated balance.) |
| - **HTP gotcha baked in**: all graph I/O is 2D `[seq,dim]` — HTP mis-lays-out 3D `[1,seq,dim]` tensors. |
| The Conv2d patch-embed and pixel-shuffle are host-side for the same reason. |
| - Source model: [`OpenGVLab/InternVL3_5-1B`](https://huggingface.co/OpenGVLab/InternVL3_5-1B) |
| (InternViT-300M + Qwen3-0.6B), exported and compiled with QAIRT 2.45 for |
| `qualcomm-snapdragon-8-elite-for-galaxy`. |
|
|
| ## v81 (SM8850 / soc_model 87) |
| |
| Device-validated on SM8850: image->caption tracks the image — e.g. an orange dahlia -> |
| *"The image features a close-up of a single, large, orange flower with a textured, layered appearance. The |
| flower has a circular shape with numerous petals radiating from a central point..."*. Vision 147 ms, prefill |
| TTFT 115 ms, decode 18.4 ms/tok. The v81 recipe was **completed** by reconstructing the two graphs the |
| `internvl_generate` host-op needs but the recipe lacked: a batched-prefill graph (`ivl_pf`, AR=512, GQA) and a |
| lm-head graph (`ivl_lmh`) — both gated vs the HF gold first token. All graphs f16 (f16 I/O), DLC route + O3. |
|
|
| ### Files (`v81/`) |
| `internvl3_5-1b.json` · `ivl_enc_f16.bin` (InternViT-300M vision) · `ivl_pf_f16.bin` (batched prefill) · |
| `ivl_dec_f16.bin` (AR=1 decode) · `ivl_lmh_f16.bin` (lm-head) · `ivl_embed_f16.bin` · `tokenizer.json` · |
| `vlm/` (host patch/pos-embed + pixel-shuffle MLP projector weights + prompt template). |
|
|
