anthonypjshaw/harrier-oss-v1-270m-qnn-htp

Harrier OSS v1 270M - Quantized for Snapdragon X Elite HTP (Hexagon NPU)

This is a static-INT8 (int8) quantized build of microsoft/harrier-oss-v1-270m, compiled by Qualcomm AI Hub to a precompiled QNN context binary wrapped in an ONNX file consumable by ONNX Runtime's QNN Execution Provider on Snapdragon X Elite (Hexagon HTP V81).

The 270M variant is the smaller, lower-latency sibling of harrier-oss-v1-0.6b built on the Gemma3-text backbone. Intended for local, on-device embedding generation.

How it was built

1. Source: PyTorch checkpoint at microsoft/harrier-oss-v1-270m (~540 MB safetensors, Gemma3-text architecture).
2. Local PT -> ONNX export via optimum.exporters.onnx.main_export( library_name="sentence_transformers", task="feature-extraction", no_dynamic_axes=True, batch_size=1, sequence_length=512, opset=17). This bakes the full Sentence-Transformers pipeline (backbone + masked mean pool + L2 normalise) into a single ONNX graph that uses only ai.onnx ops and contains no GatherND nodes.
3. Local graph cleanup: SymbolicShapeInference + onnxsim.simplify to fold constants and propagate shape info before upload (AI Hub's converter env doesn't ship with onnxruntime or onnxsim).
4. AI Hub compile: a single submit_compile_job call with: - --target_runtime precompiled_qnn_onnx - --quantize_full_type int8 (static INT8 weights + activations) - --truncate_64bit_io (HTP I/O is bounded to int32) - calibration_data = 20 tokenized representative natural-language inputs targeting Snapdragon X Elite CRD running OS 11.

Because the PT -> ONNX export is done locally with optimum's sentence-transformers OnnxConfig (not AI Hub's dynamo converter), the resulting graph reaches the Hexagon NPU end-to-end without the GatherND op that previously broke QNN's CPU validator.

Files

File	Purpose
harrier-oss-v1-270m.qnn_ctx.onnx	ONNX wrapper containing the embedded QNN HTP context binary
tokenizer.json	HF Tokenizers JSON (copied verbatim from the source repo)
tokenizer_config.json, special_tokens_map.json	Tokenizer metadata
aihub-metadata.txt	URLs of the AI Hub compile / profile jobs

Usage (Node.js / ONNX Runtime)

import * as ort from "onnxruntime-node";

const session = await ort.InferenceSession.create(
  "harrier-oss-v1-270m.qnn_ctx.onnx",
  {
    executionProviders: [
      {
        name: "qnn",
        backend_path: "QnnHtp.dll",
        htp_performance_mode: "burst",
        htp_graph_finalization_optimization_mode: "3",
      },
    ],
    graphOptimizationLevel: "all",
  },
);

const inputIds = new BigInt64Array(512).fill(0n);
const attentionMask = new BigInt64Array(512).fill(1n);
const feeds = {
  input_ids: new ort.Tensor("int64", inputIds, [1, 512]),
  attention_mask: new ort.Tensor("int64", attentionMask, [1, 512]),
};
const { embedding } = await session.run(feeds);

When to pick this vs the 0.6B build

Concern	270M (this)	0.6B
Latency on HTP	Lower (smaller graph)	Higher
Embedding quality	Lower (smaller model)	Higher
Peak memory	Lower	Higher
Best for	Always-on background tasks, embedded scenarios	Foreground retrieval where quality matters

Limitations

• Fixed sequence length: 512 tokens. Pad or truncate accordingly.
• SoC-specific: built for HTP V81 (Snapdragon X Elite). Other Snapdragon SoCs may need a separate compile job.
• Quantization loss: typical for INT8 static. Expect a small drop in retrieval recall vs the FP32 baseline; if accuracy matters more than latency, use the upstream FP32 model on CPU or step up to the 0.6B build.

License

MIT, inherited from microsoft/harrier-oss-v1-270m.