xpuenabler/molmoact2-libero_grid_sampler_qnn-visfused-actstepfused

MolmoAct2-LIBERO — Operator-Fused vision + action_step (QNN / HTP context binaries)

Operator-fused variants of the vision and action_step components of allenai/MolmoAct2-LIBERO, converted to Qualcomm QNN (HTP) context binaries for Dragonwing IQ-9075 (QCS9075, Hexagon v73, soc_id 77).

This repo is a companion to xpuenabler/molmoact2-libero_grid_sampler_qnn: it ships only the fused artifacts (fused ONNX + newly-built HTP context binaries) for the two modules where a projection-level fusion applies. The fused context binaries keep the same graph names (vision, action_step) and the same I/O contract as the unfused ones, so they are drop-in replacements in the resident runtime — the remaining components (llm_split0..3, action_context) are unchanged and taken from the base repo.

Model variant / sequence length. These fused graphs are the original allenai/MolmoAct2-LIBERO path: prompt seq = 488, vision emits 392 image tokens (2 crops × 196). This differs from the grid-sampler (seq=128 / 32-token) golden shipped in the base repo — see Verification for why parity is checked by ONNX-equivalence here.

---

What's in here

onnx/
  vision_fused/         vision.onnx + vision_weights.bin (1.76 GB) + fusion_info.json
  action_step_fused/    action_step.onnx + action_step_weights.bin (2.30 GB) + fusion_info.json
  operator_fusion_applied.md    what was fused + numerical-equivalence verification
  operator_fusion_report.md     fusion-opportunity survey across all modules
  fusion_viz/                   netron before/after screenshots of the fused subgraphs
ctx/      7 HTP context binaries (fp16, soc_id 77 / Hexagon v73) — a complete resident set
  vision_socid77_archv73.bin        931 MB  — FUSED (this repo)
  action_step_socid77_archv73.bin  1160 MB  — FUSED (this repo)
  llm_split0_socid77_archv73.bin   1820 MB  — unchanged (from base repo)
  llm_split1_socid77_archv73.bin   1820 MB  — unchanged
  llm_split2_socid77_archv73.bin   1820 MB  — unchanged
  llm_split3_socid77_archv73.bin   1820 MB  — unchanged
  action_context_socid77_archv73.bin  3.6 MB  — unchanged
golden/
  vision_io.npz  action_step_io.npz   reference I/O carried from the base repo
README.md

ctx/ is a complete 7-session resident set: the two fused binaries (vision, action_step) built here, plus the unchanged llm_split0..3 and action_context carried over from the base repo. All graph names are unchanged, so this drops straight into the base repo's runtime/.

---

What was fused

Parallel projections sharing the same input activation are weight-concatenated into a single MatMul followed by a Split that restores the original per-branch output tensor names — so bias Add, reshape, and activation subgraphs are untouched and Split(concat(W)) ≡ [W_i] holds bit-exactly. (Details + before/after node counts in onnx/operator_fusion_applied.md.)

module	fusion	targets	weight change
vision	QKV (wq+wk+wv → wqkv)	ViT resblocks ×25	[1152,1152]×3 → [1152,3456]
vision	gate_up (w1+w3 → gate_up, SwiGLU)	image_projector ×1	[1152,9728]×2 → [1152,19456]
vision	KV (wk+wv → kv)	image_pooling_2d ×1	[2304,1152]×2 → [2304,2304]
action_step	gate_up (gate_proj+up_proj → gate_up)	DiT blocks ×36	[768,3072]×2 → [768,6144]

Node-count deltas: vision MatMul 210→158 (Split 0→27); action_step MatMul 398→362 (Split 0→36). The fused ONNX are numerically identical to the originals (max_abs_diff = 0.000e+00, onnxruntime, ORT_DISABLE_ALL).

---

Verification

The base repo's golden tensors are the grid-sampler (seq=128) variant, whereas these fused graphs are the original seq=488 path — so those golden tensors cannot serve as a PyTorch parity reference here. Instead, each fused DLC is verified against its fused ONNX on identical inputs (real golden pixel_values for vision; deterministic synthesized inputs at the ONNX's declared shapes for action_step), run on the QNN CPU backend vs onnxruntime:

context	output	DLC ≡ ONNX cosine	within_tol
vision	image_embeds [392,2560]	1.00000000	1.0000
action_step	next_trajectory [1,10,32]	1.00000000	1.0000

Chain of trust: fused DLC ≡ fused ONNX (above) and fused ONNX ≡ original ONNX ≡ PyTorch (upstream max_abs_diff = 0). fp16-on-HTP numerics + latency are validated on device by the base repo's runtime/.

---

Build provenance

• Base model: allenai/MolmoAct2-LIBERO (original, non-grid-sampler; prompt seq = 488).
• Fusion: onnx 1.18.0, projection weight-concat + Split (numerically exact); see onnx/.
• Toolchain: QAIRT 2.47.0, opset 20, float DLC (no quant), offline HTP context-binary generation for soc_id 77 / dsp_arch v73, O3.
• Path: fused ONNX → qairt-converter DLC → CPU-backend equivalence check → qnn-context-binary-generator (.bin).
• Graph names unchanged (vision, action_step) → drop-in for the resident runtime.