README.md

---
license: apache-2.0
base_model: ZibinDong/fastwam_libero_uncond_2cam224
tags:
- fast-wam
- world-model
- mot
- vla
- robotics
- libero
- qnn
- qualcomm
- htp
- hexagon
- npu
- edge
pipeline_tag: robotics
---

# Fast-WAM (LIBERO) — QNN / HTP context binary (action expert)

Offline-compiled **Qualcomm QNN (HTP) context binary** for the MoT **action expert** of
[`ZibinDong/fastwam_libero_uncond_2cam224`](https://huggingface.co/ZibinDong/fastwam_libero_uncond_2cam224),
ready to run **resident on-device** on a **Qualcomm Dragonwing IQ‑9075 (QCS9075, Hexagon v73, soc_id 77)**.

This is the **runtime deployment bundle**: the prebuilt `action_step` `.bin` context, the golden
reference tensors (inputs + parity reference, including the video KV cache), and the resident
runtime. The conversion pipeline (`PyTorch → ONNX → QNN DLC → HTP context binary`) lives in the
[`nota-github/xpu-molmoact2-qnn-htp`](https://github.com/nota-github/xpu-molmoact2-qnn-htp) repo
(`fastwam_qnn/`, branch `support_Fast-WAM`, PR #2).

> **What's on the NPU.** Fast-WAM is a Mixture-of-Transformers (MoT) world model: a frozen Wan2.2
> video VAE + UMT5 text encoder feed a **video DiT** (world model) and an **action DiT** (action
> expert). This bundle ships the **action expert** denoise step — after a **real-valued-RoPE +
> constant-mask surgery** it converts to HTP with **cosine 1.000000** CPU-backend parity. The
> **video world-model prefill** (which produces the video KV cache the action expert cross-attends
> to) converts to a QNN DLC with cosine 1.0 too, but its fp16 context is ~9.7 GB (> the device's
> ~2 GB/session limit) so it needs a MolmoAct2-style layer-split before it can go on-NPU; for now
> it runs **host-side** and the runtime feeds its output (the golden video KV cache). The frozen
> Wan2.2 VAE stays host-side (Conv3d 5D exceeds QNN's rank-5 limit). See [Remaining](#remaining).

---

## What's in here

```
ctx/      1 HTP context binary (soc_id 77 / Hexagon v73), fp16 weights
  action_step_socid77_archv73.bin       2.06 GB
golden/   reference I/O (PyTorch fp32): action-step inputs (incl. the video KV cache) + parity ref
  action_step_io.npz     video_prefill_io.npz     trace.json     convert_report.json
runtime/  resident runtime
  resident_run.py      orchestrator (host glue: flow-match Euler loop, parity, latency)
  resident_worker.py   one NPU session per process (loads the action_step context once)
  profile_device.sh    one-command host→device profiler (SSH; reads creds from IQ9_info.txt)
```

You also need, **from the QAIRT 2.47.x SDK** (not redistributed here):
`qnn_libs/` (aarch64 QNN runtime `.so`s) and `dsp_libs/` (Hexagon **v73** skel libs).

---

## Architecture — 1 on-NPU component

```
  image ─┐  Wan2.2 VAE      ┌───────────────┐  video KV cache   ┌────────────────────┐
         ├───(host)────────▶│  video DiT    │ ────────────────▶ │  action DiT (NPU)  │─▶ actions
 text ───┘  UMT5 (host)     │ world model   │  [30,1,98,3072]   │  flow-matching     │   [1,32,7]
                            │ prefill (host)│  (×2 k/v)         │  step ×N (resident)│
                            └───────────────┘                   └────────────────────┘
```

- The **action expert** (`action_step`) is a single HTP context, **looped N× on the host** for
  flow-matching denoising. Only `latents_action` and `timestep` change across the loop; the video
  KV cache + context are constant per chunk.
- The **video world-model prefill** and the **VAE** run host-side for now (see [Remaining](#remaining));
  the runtime feeds their outputs from `golden/`.

### I/O contract

| context | inputs (name: shape, f32) | output (name: shape, f32) |
|---|---|---|
| `action_step` | `latents_action: [1,32,7]`, `timestep: [1]`, `video_keys: [30,1,98,3072]`, `video_values: [30,1,98,3072]`, `context: [1,129,4096]`, `context_mask: [1,129]` | `pred_noise: [1,32,7]` |

Host loop (Wan continuous flow-match): schedule `sigma = linspace(1,0,N+1)[:N]; sigma = shift·sigma/(1+(shift−1)·sigma)`,
`timesteps = sigma·1000`, `deltas = sigma_next − sigma`; for step `i`, feed `latents_action` + `timesteps[i]`,
take `pred_noise`, update `latents_action += pred_noise · deltas[i]`. (shift = 5.0.)

---

## Inference guide

### 1. Device workspace

On the IQ‑9075, create `/root/fastwam_workspace/` with:

```
/root/fastwam_workspace/
  ctx/        ← action_step_socid77_archv73.bin from this repo
  golden/     ← the golden/ folder from this repo
  runtime/    ← resident_run.py, resident_worker.py  (profile_device.sh pushes these for you)
  qnn_libs/   ← aarch64 QNN runtime .so from QAIRT 2.47.x: lib/aarch64-oe-linux-gcc11.2/*.so
  dsp_libs/   ← Hexagon v73 skel from QAIRT 2.47.x:        lib/hexagon-v73/unsigned/*.so
  tmp/        ← created automatically
```

The runtime loads them via `LD_LIBRARY_PATH=$WS/qnn_libs` and `ADSP_LIBRARY_PATH=$WS/dsp_libs`.
Device Python needs **`qai_appbuilder`** (QAI AppBuilder for QNN) + `numpy`.

> **action_step is 2.06 GB** — right at the device's ~2 GB single-session fastrpc weights-buffer
> limit. **Reboot before a clean resident run** (crashed cycles leak DSP sessions); the worker ends
> with `os._exit()` to dodge an appbuilder↔libs teardown double-free.

### 2a. Run directly on the device

```bash
cd /root/fastwam_workspace
python3 runtime/resident_run.py          # FASTWAM_STEPS=10 (default), FASTWAM_SHIFT=5.0
```

It spawns the worker, keeps it **resident**, runs the N-step flow-match loop, prints the
**single-step pred cosine** vs the golden reference, then a latency summary (avg of 5) plus the
pure per-step `[infer-ms]` (excl. TCP) in `tmp/w_action_step.log`.

### 2b. Or profile from the host (one command)

```bash
runtime/profile_device.sh
```

Copies the runtime scripts to an ephemeral `/tmp` workspace, symlinks the heavy assets, runs, prints
the latency table + per-step `[infer-ms]`, then deletes the temp dir. Device creds are read from
`IQ9_info.txt` (`IP:` / `passwd:`; override with `IQ9_INFO=...`).

### fp16 / quantized A/B switch

```bash
ACTION_CTX_SUFFIX=_w4a16 python3 runtime/resident_run.py   # if you add a quantized bin
```

---

## Validated parity (host, QNN CPU backend vs PyTorch fp32)

Verified device-free on the QNN **CPU** backend against the PyTorch golden. **Cosine is the gate**
(≥ 0.9999).

| stage | output | cosine |
|---|---|---|
| ONNX (ORT CPU) — reference | — | action expert exports clean after real-RoPE surgery |
| QNN DLC (CPU backend) vs PyTorch | pred_noise | **1.000000** |
| video_prefill DLC (CPU backend) vs PyTorch | video_keys / video_values | 1.000000 / 1.000000 |

The on-device (HTP fp16) single-step pred cosine + latency are what `resident_run.py` reports.

---

## Remaining

- **video world-model prefill — layer-split.** The 30-layer video DiT converts to a QNN DLC with
  cosine 1.0, but its fp16 context (~9.7 GB) exceeds the device's ~2 GB/session limit. Split it into
  layer-range contexts (MolmoAct2-LLM pattern) to run on-NPU; until then the runtime uses the
  host-computed video KV cache from `golden/`.
- **Wan2.2 VAE — host-side.** The frozen `AutoencoderKLWan` is Conv3d (5D); an internal transpose
  exceeds QNN HTP's rank-5 limit. Keep it host-side (feed latents), or refactor the T=1 path to rank-4.

---

## Build provenance

- Source policy: [`ZibinDong/fastwam_libero_uncond_2cam224`](https://huggingface.co/ZibinDong/fastwam_libero_uncond_2cam224).
- Action expert: MoT action DiT (30 layers, hidden 1024, 24 heads × 128), flow-matching, action chunk `[1,32,7]`.
- Surgery for ONNX/HTP: complex `view_as_complex`/`view_as_real` RoPE → real rotate-half (`.freqs`
  stored as real cos/sin); MoT boolean attention masks precomputed as constant buffers.
- Toolchain: QAIRT **2.47.0**, opset 20, float DLC (no quant), offline HTP context-binary for
  soc_id 77 / dsp_arch v73, `O3`.
- Conversion code: `nota-github/xpu-molmoact2-qnn-htp` → `fastwam_qnn/`, `scripts_fastwam/`, `fastwam_qnn/surgery.py` (branch `support_Fast-WAM`, PR #2).