anthonypjshaw/harrier-oss-v1-0.6b-qnn-htp

Harrier OSS v1 0.6B - Quantized for Snapdragon X Elite HTP (Hexagon NPU)

This is a static-INT8 (int8) quantized build of microsoft/harrier-oss-v1-0.6b, compiled by Qualcomm AI Hub to a precompiled QNN context binary wrapped in an ONNX file consumable by ONNX Runtime's QNN Execution Provider on Snapdragon X Elite (Hexagon HTP V81).

How it was built

1. Source: PyTorch checkpoint at microsoft/harrier-oss-v1-0.6b (1.2 GB safetensors).
2. TorchScript trace: torch.jit.trace at static shape (batch=1, seq_len=512). The traced module wraps the backbone with masked mean pooling over the last hidden state and L2-normalises, so the exported model behaves as a sentence encoder returning a unit-norm embedding.
3. AI Hub compile: a single submit_compile_job call with: - --target_runtime precompiled_qnn_onnx - --quantize_full_type int8 (static INT8 weights + activations) - --truncate_64bit_io (HTP I/O is bounded to int32) - calibration_data = 20 tokenized representative natural-language inputs AI Hub internally performs PT -> ONNX conversion, PTQ, and HTP lowering for Snapdragon X Elite CRD running OS 11.

Because the conversion happens inside AI Hub's own translator, the resulting ONNX graph uses only ai.onnx ops - none of the com.microsoft.* contrib ops that the pre-fused onnx-community/harrier-oss-v1-0.6b-ONNX export contains - so the graph reaches the Hexagon NPU end-to-end without CPU fallback.

Files

File	Purpose
harrier-oss-v1-0.6b.qnn_ctx.onnx	ONNX wrapper containing the embedded QNN HTP context binary
tokenizer.json	HF Tokenizers JSON (copied verbatim from the source repo)
tokenizer_config.json, special_tokens_map.json	Tokenizer metadata
aihub-metadata.txt	URLs of the AI Hub compile / profile jobs

Usage (Node.js / ONNX Runtime)

import * as ort from "onnxruntime-node";

const session = await ort.InferenceSession.create(
  "harrier-oss-v1-0.6b.qnn_ctx.onnx",
  {
    executionProviders: [
      {
        name: "qnn",
        backend_path: "QnnHtp.dll",
        htp_performance_mode: "burst",
        htp_graph_finalization_optimization_mode: "3",
      },
    ],
    graphOptimizationLevel: "all",
  },
);

const inputIds = new BigInt64Array(512).fill(0n);
const attentionMask = new BigInt64Array(512).fill(1n);
const feeds = {
  input_ids: new ort.Tensor("int64", inputIds, [1, 512]),
  attention_mask: new ort.Tensor("int64", attentionMask, [1, 512]),
};
const { embedding } = await session.run(feeds);

Limitations

• Fixed sequence length: 512 tokens. Pad or truncate accordingly.
• SoC-specific: built for HTP V81 (Snapdragon X Elite). Other Snapdragon SoCs may need a separate compile job.
• Quantization loss: typical for INT8 static. Expect a small drop in retrieval recall vs the FP32 baseline; if accuracy matters more than latency, use the upstream FP32 model on CPU.

License

MIT, inherited from microsoft/harrier-oss-v1-0.6b.