justinchuby/qwen2.5-0.5b-instruct-onnx-genai

Qwen2.5-0.5B-Instruct for onnx-genai

An INT4 ONNX package that runs directly with the nxrt Python API. It demonstrates onnx-genai's native inference_metadata.yaml format, which is the authoritative decoder contract in this package. genai_config.json is retained only for ecosystem compatibility.

The metadata names every actual decoder port, including all 24 key/value cache pairs. This graph does not expose a position_ids input, so none is declared.

Run

python3 -m pip install nxrt
ONNX_GENAI_KV_MAX_LEN=128 python3 - <<'PY'
import nxrt

engine = nxrt.genai.Engine.from_dir(".")
result = engine.generate(
    "Hello! In one sentence, what is Rust?",
    max_tokens=16,
    temperature=0.0,
)
print(result.text)
print(result.token_ids)
print(result.finish_reason)
PY

Verified with nxrt==0.1.0.dev3 on CPU:

 Rust is a high-level programming language designed to be efficient and easy to use.
[33789, 374, 264, 1550, 11591, 15473, 4128, 6188, 311, 387, 11050, 323, 4135, 311, 990, 13]
max_tokens

ONNX_GENAI_KV_MAX_LEN limits the shared KV allocation for this short example; the model metadata preserves the full 32,768-token context limit.

Attribution and license

Derived from Qwen/Qwen2.5-0.5B-Instruct. Qwen2.5 is distributed under the Apache License 2.0. See LICENSE.