baulab/Qwen3-VL-2B-Instruct-GazeHeads-ONNX

Qwen3-VL-2B-Instruct — Gaze-Head Steering (ONNX, WebGPU)

ONNX build of Qwen3-VL-2B-Instruct (via onnx-community's export) with two extra decoder inputs that expose the gaze-head steering intervention from the paper Gaze Heads: How VLMs Look at What They Describe:

• gaze_head_mask (28, 16) float16 — 1.0 selects an attention head
• gaze_sign (total_sequence_length,) float16 — additive pre-softmax attention bias per KV position (e.g. +10000 on target image tokens, -10000 on other image tokens, 0 on text)

Feeding zeros reproduces the original model. The layers hosting the top-20 gaze heads use decomposed attention (plain ops) so the bias works on the onnxruntime-web WebGPU backend, whose fused GroupQueryAttention kernel ignores attention_bias; all other layers keep the fused kernel.

vision_encoder_q4f16.onnx is a stub: the demo injects precomputed image features and never runs the vision tower. Use the onnx-community repo if you need the real encoder.

Only the decoder graph file differs from the upstream export; all weight data files are bit-identical copies.

Demo / paper / code: see the project page.