Publish parse Hexagon-NPU bundle (v81) + model card

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+v81/doc_sample.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,116 @@
+---
+license: other
+license_name: nvidia-open-model-license
+license_link: https://huggingface.co/nvidia/NVIDIA-Nemotron-Parse-v1.2
+tags:
+  - qhexrt
+  - hexagon
+  - npu
+  - qnn
+  - vlm
+  - ocr
+  - document-understanding
+base_model: nvidia/NVIDIA-Nemotron-Parse-v1.2
+pipeline_tag: image-text-to-text
+---
+
+# Nemotron-Parse-v1.2 — Hexagon NPU (QHexRT)
+
+Document OCR + layout parse (`nvidia/NVIDIA-Nemotron-Parse-v1.2`) compiled to run on the Qualcomm
+Hexagon NPU through the [QHexRT](https://github.com/) runtime. Give it a document image; it returns the
+text plus layout markup (`<class_Section-header>` / `<class_Text>` …) and bounding-box tokens.
+
+It runs as a QHexRT **AED-VLM** plan: a RADIO ViT-H vision encoder feeds an MBart attention-encoder-decoder
+that greedily decodes the page. Only the two heavy graphs (vision, decoder) run on the NPU; the glue is host C++.
+
+## Arch
+
+- **Hexagon v81** (`dsp_arch=v81`, `soc_model=87`). Bundle dir: `v81/`.
+- **Arch-pinned:** a context binary bakes in `dsp_arch` + `soc_model`, so these `.bin`s load **only** on v81.
+  A different Hexagon arch needs a re-compile into a sibling `v79/` / `v83/` dir (load failure, not wrong output).
+- Precision: **fp16** on both graphs (no quantization yet; W8A16 weight-only is the future perf/memory lever).
+
+## Pipeline — what runs where
+
+| stage | where | detail |
+|---|---|---|
+| RADIO preprocess | host C++ | CLIP mean/std normalize, `im_to_patches` 16×16, embedder 768→1280, 8 prefix (cls/register) tokens, CPE pos-embed → `embed[3336,1280]` |
+| **vision encoder** | **NPU graph** `radio_enc_1024` | C-RADIOv2-H, 32-block ViT-H (1280-dim, 16 heads), full attention. `embed[3336,1280] → vit_feat[3336,1280]`, fp16, 2D I/O |
+| neck | host C++ | drop 8 prefix → conv1→LN → window conv2→LN + summary token → **833 cross-attn memory tokens** `mem[833,1024]` |
+| cross-KV | host C++ | project `mem` by each decoder layer's `encoder_attn.{k,v}_proj` → the cross-attention K/V, bound once |
+| **decoder step** | **NPU graph** `parse_decoder_step` | MBart AED, 10 layers, d1024/16 heads. Single-step **fixed-MDL=512 KV cache**, in-graph int32 embed + in-graph untied lm-head → `logits[52352]`, fp16. No `position_ids` (MBart has no positional embedding) |
+| decode loop | host C++ | greedy argmax, self-KV ping-pong, seed = eos = `decoder_start_token_id` = 2, stop on predicted eos → detok → text + layout markup |
+
+No custom op-package (RADIO ViT + AED decoder are native QNN ops) → **no `ADSP_LIBRARY_PATH` needed**.
+
+## Files (`v81/`)
+
+| file | role | size |
+|---|---|---|
+| `nemotron-parse.json` | QHexRT manifest (the declarative run plan) | 2.4 KB |
+| `radio_enc_1024.bin` | vision encoder context binary (RADIO ViT-H, fp16) | 1329.1 MB |
+| `parse_decoder_step.bin` | MBart AED decoder-step context binary (KV-cache + in-graph lm-head, fp16) | 512.7 MB |
+| `parse_radio_preproc.bin` | host blob: input-conditioner + patch-generator weights (CLIP norm, embedder 768→1280, prefix, CPE pos) | 21.0 MB |
+| `parse_neck.bin` | host blob: neck weights (conv1, ln1, conv2, ln2, sum_proj, ln3) | 44.3 MB |
+| `parse_cross_proj.bin` | host blob: `encoder_attn.{k,v}_proj` weight+bias ×10 layers (cross-KV projection) | 84.0 MB |
+| `tokenizer.json` | tokenizer (vocab 52352) | 4.0 MB |
+| `doc_sample.png` | device-gate fixture document | 0.1 MB |
+
+Total ≈ **2.0 GB**. The QNN runtime libs (`libQnnHtp.so`, `libQnnSystem.so`, the v81 HTP skel) come from the
+QAIRT SDK — they are **not** in this repo.
+
+## Run (on-device, over adb)
+
+Stage the QAIRT aarch64 runtime libs + the `qhx_generate` tool into `/data/local/tmp/wq` once (see the QHexRT
+`docs/DEPLOY.md`), then:
+
+```bash
+hf download runanywhere/nemotron_parse_HNPU --local-dir nemotron_parse_HNPU
+adb push nemotron_parse_HNPU/v81 /data/local/tmp/wq/nemotron_parse   # PowerShell + native paths on Windows
+
+adb shell "cd /data/local/tmp/wq && LD_LIBRARY_PATH=. \
+  ./qhx_generate nemotron_parse/nemotron-parse.json libQnnHtp.so libQnnSystem.so nemotron_parse \
+  512 '' nemotron_parse/doc_sample.png"
+```
+
+Tool arg order is invariant: `<tool> <manifest> libQnnHtp.so libQnnSystem.so <artifacts_root> <max_new> "<prompt>" [image]`.
+The **image is the input**; the text prompt is ignored (the host-op runs its own greedy decode to eos). `max_new`
+caps the decode at the MDL (512 → 511 tokens ≈ a paragraph / simple page).
+
+## Device validation (Hexagon v81, S25 reference device)
+
+Validated per the QHexRT three-gate rubric (offline oracle → on-device raw-I/O parity → end-to-end over multiple inputs):
+
+- **Vision, on-device raw-I/O parity:** `embed[3336,1280]` F16 → `vit_feat`, cosine **0.999030** vs the offline
+  reference at RES=1024 (3336-token ViT). (Offline oracle cos 1.0007 vs `vit.forward_features`.)
+- **Decoder:** offline fixed-MDL(512) step cosine **1.000001** vs the full-seq `NemotronParseDecoder`; device-functional
+  (loads, emits `logits[52352]` + all self-KV out — the KV ring works).
+- **End-to-end** on `doc_sample.png` → reads the document text **exactly** vs the pure-greedy gold (65-token chain,
+  seed 2 → eos 2): title + body text + `<class_Section-header>`/`<class_Text>` layout markup + bbox tokens. Confirmed
+  on **3 additional synthetic documents**.
+- Text content is **exact**; bounding-box y-coordinates drift ~0.03 (fp16 coordinate-regression precision, a
+  `greedy_tol`-class tolerance — not a text error).
+
+### Measured performance (v81)
+
+| phase | time |
+|---|---|
+| vision encoder (3336-wide self-attention) | ~1986 ms |
+| decode | ~14 ms/token |
+| end-to-end (`doc_sample.png`, 65 tokens) | ~15.7 s |
+
+## Caveats
+
+- **Fixed device resolution 1024×832.** The processor resizes by longest edge then white-pads; any-aspect document
+  is letterboxed to 1024×832 on the host (patch grid stays 64×52). Other resolutions need a re-export.
+- **Output cap ≈ 511 tokens** (MDL=512) — a paragraph or a simple page. A full multi-page document needs MDL raised
+  and the decoder re-exported.
+- **Memory:** the 3336-wide vision self-attention spills ~5.5 GB DDR, so a ~12 GB device (S25) is required.
+- **fp16, not quantized.** Bounding-box coordinates are the only place fp16 shows (~0.03 drift); text is exact.
+- Greedy (temp=0) is the parity metric; the source config's `repetition_penalty=1.1` is a sampler-quality knob,
+  identical to pure greedy on short documents.
+
+## License
+
+Governed by the source model's license (`nvidia/NVIDIA-Nemotron-Parse-v1.2`, NVIDIA Open Model License). This repo
+ships only the compiled Hexagon context binaries + host weights + manifest, not the original checkpoints.

v81/nemotron-parse.json

@@ -0,0 +1,35 @@
+{
+  "schema_version": 1,
+  "_comment": "NB-9 NVIDIA-Nemotron-Parse-v1.2 document OCR / layout-parse as a QHexRT AED-VLM family (image -> RADIO ViT-H vision -> host neck -> MBart AED decoder -> autoregressive greedy). The nemotron_parse_generate host-op runs the whole pipeline: host RADIO preprocess (LongestMaxSize-resize + white-pad to 1024x832, /255, OPENAI-CLIP norm, patch_generator im_to_patches 16x16 -> embedder 768->1280 + 8 prefix + CPE pos_embed) -> NPU vision graph radio_enc_1024 (32 RADIO blocks, 2D I/O embed[3336,1280] -> vit_feat[3336,1280], fp16) -> host neck (drop 8 prefix -> conv1/LN1 -> window conv2/LN2 + summary token -> 833 cross-attn memory tokens[833,1024]) -> host cross-KV (mem @ decoder.encoder_attn.{k,v}_proj per layer) -> NPU decoder step graph parse_decoder_step (single-step fixed-MDL=512 KV cache: input_ids[1] int32 in-graph embed, attention_mask[512], k/v_cache_self_N_in/out[512,1024], k/v_cache_cross_N[833,1024], in-graph lm-head -> logits[52352]) greedy decode loop (asr_whisper pattern: seed=eos=decoder_start_token_id=2, right-aligned mask, self-KV ping-pong, NO position_ids). No custom op-package (RADIO ViT + AED decoder are native QNN ops) -> no ADSP_LIBRARY_PATH needed (like Whisper). Host weight blobs (parse_radio_preproc.bin, parse_neck.bin, parse_cross_proj.bin) + tokenizer.json in fixture_dir. Offline-gated: preprocess cos 1.0008, neck cos 1.000066, cross-KV cos 1.000000; both graphs device-functional (vision Gate-B cos 0.999030, decoder loads + emits logits). Graph names are the compiled binary's real internal names (radio_enc_1024_f16 / parse_decoder_step_f16 from build_f16.sh); confirm with qhx_meta after compile.",
+  "model": { "name": "nemotron-parse", "family": "vlm", "dsp_arch": "v79" },
+  "params": {
+    "vocab": 52352,
+    "n_layers": 10,
+    "hidden": 1024,
+    "head_dim": 64,
+    "max_ctx": 512,
+    "eos_token_id": 2
+  },
+  "artifacts": {
+    "contexts": {
+      "vision": { "bin": "radio_enc_1024.bin" },
+      "decoder": { "bin": "parse_decoder_step.bin" }
+    },
+    "tokenizer": "tokenizer.json",
+    "fixture_dir": "."
+  },
+  "plan": {
+    "steps": [
+      {
+        "host": "nemotron_parse_generate",
+        "params": {
+          "vision": "radio_enc_1024_f16",
+          "decoder": "parse_decoder_step_f16",
+          "res": 1024,
+          "decoder_start": 2,
+          "eos": 2
+        }
+      }
+    ]
+  }
+}

v81/parse_cross_proj.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:99c064845844826b7adb7caa3a216bb9fb691e7a69fd8ceb3fe1802d1d198be8
+size 83968000

v81/parse_decoder_step.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ae6924eba1bfe6b4d215999a0ae9383b2a50a5c7c664d4e3fff386ff73720be5
+size 512659456

v81/parse_neck.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:066c5afabfb38130fda03195a8b8ad389a797fb97a75a7c89d430b976649d858
+size 44340240

v81/parse_radio_preproc.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53d17ca48c9b709453ad59546fa062f0afe79c0a2d5d078e836daec7a01ece21
+size 21017600

v81/radio_enc_1024.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e9287a8f01692ef6c3f9a3b5ee38abe526028ea7b7df927c93c8df2dd8a3b6c7
+size 1329102848