Morteza89/gpt-oss-120b-int8-ov

GPT-OSS 120B OpenVINO INT8

This repository contains an unofficial OpenVINO™ IR conversion of openai/gpt-oss-120b with INT8 weight compression.

The model is intended for local text generation using OpenVINO GenAI. OpenVINO GenAI provides an optimized generative inference API for OpenVINO models and is the recommended runtime path here for getting strong performance on Intel hardware, including Intel CPUs and supported Intel GPUs.

Original model

• Original model: openai/gpt-oss-120b
• Original creator: OpenAI
• License: Apache 2.0
• Converted format: OpenVINO IR
• Weight format: INT8
• Task: text generation

This is an unofficial converted model repository. Please refer to the original model card for full model details, intended use, safety notes, license terms, and limitations.

Why OpenVINO GenAI?

This repository is designed to run with openvino-genai, specifically openvino_genai.LLMPipeline.

OpenVINO GenAI is optimized for generative AI inference with OpenVINO. Its LLM pipeline can load an OpenVINO-converted model folder directly, including the model, tokenizer, detokenizer, and generation configuration. This makes it a clean and efficient runtime path for local LLM inference on Intel hardware.

Quantization

• Weight format: INT8
• Export tool: Optimum Intel
• Runtime: OpenVINO GenAI
• Model format: OpenVINO IR

INT8 compression is intended to reduce model size and memory usage compared with higher precision weights while typically preserving more numerical fidelity than more aggressive lower-bit compression. As with any converted and quantized model, quality and numerical behavior may differ from the original model and should be validated for your use case.

Conversion

This model was converted with Optimum Intel using the OpenVINO export path.

optimum-cli export openvino
--model openai/gpt-oss-120b
--weight-format int8
--trust-remote-code
ov_gpt_oss_120b_int8

Example tested command

python test_gpt_oss_ov_genai.py
--model-dir ov_gpt_oss_120b_int8
--device CPU
--max-new-tokens 256

Repository contents

Key files include:

chat_template.jinja
config.json
generation_config.json
openvino_config.json
openvino_model.xml
openvino_model.bin
openvino_tokenizer.xml
openvino_tokenizer.bin
openvino_detokenizer.xml
openvino_detokenizer.bin
special_tokens_map.json
tokenizer.json
tokenizer_config.json
examples/run_gpt_oss_openvino_genai.py
examples/requirements.txt
python examples/run_gpt_oss_openvino_genai.py \
  --model-dir . \
  --device CPU \
  --max-new-tokens 256 \
  --prompt "Explain quantum mechanics clearly and concisely."