Morteza89/qwen2-57b-a14b-int4-ov

Qwen2 57B-A14B OpenVINO INT4

This repository contains an unofficial OpenVINO™ IR conversion of Qwen/Qwen2-57B-A14B with INT4 weight compression.

The model is intended for local text generation using OpenVINO GenAI. OpenVINO GenAI provides an optimized generative inference API for OpenVINO models and is the recommended runtime path here for getting strong performance on Intel hardware, including Intel CPUs and supported Intel GPUs.

Original model

• Original model: Qwen/Qwen2-57B-A14B
• Model family: Qwen2
• Architecture: Qwen2 MoE causal language model
• Converted format: OpenVINO IR
• Weight format: INT4
• Task: text generation

This is an unofficial converted model repository. Please refer to the original model card for full model details, intended use, license terms, safety notes, and limitations.

Model summary

Qwen2-57B-A14B is a Mixture-of-Experts language model from the Qwen2 family. Qwen2 models are decoder language models based on Transformer architecture, with features such as SwiGLU activation, attention QKV bias, group query attention, and an improved tokenizer designed for multiple natural languages and code.

This OpenVINO version is designed for efficient local inference with OpenVINO GenAI.

Why OpenVINO GenAI?

This repository is designed to run with openvino-genai, specifically openvino_genai.LLMPipeline.

OpenVINO GenAI is optimized for generative AI inference with OpenVINO. Its LLM pipeline can load an OpenVINO-converted model folder directly, including the model, tokenizer, detokenizer, chat template, and generation configuration. This makes it a clean and efficient runtime path for local LLM inference on Intel hardware.

Conversion

This model was converted with Optimum Intel using the OpenVINO export path.

optimum-cli export openvino \
  --model Qwen/Qwen2-57B-A14B \
  --weight-format int4 \
  --trust-remote-code \
  ov_Qwen2_57B_A14B_int4
Quantization
Weight format: INT4
Export tool: Optimum Intel
Runtime: OpenVINO GenAI
Model format: OpenVINO IR

INT4 compression is intended to reduce model size and memory usage compared with higher precision weights. As with any converted and quantized model, quality and numerical behavior may differ from the original model and should be validated for your use case.
Usage
Run from inside the model directory
python examples/run_qwen2_openvino_genai.py \
  --model-dir . \
  --device CPU \
  --max-new-tokens 128 \
  --prompt "Explain what OpenVINO is in one short paragraph."
  
Limitations

This repository inherits the limitations of the original Qwen/Qwen2-57B-A14B model. Additional differences may arise from OpenVINO conversion, INT4 compression, and runtime configuration.

Attribution

This is an unofficial OpenVINO conversion of the original Qwen2 model. All rights to the original model, training, and licensing remain with the original authors.

## Serve with OpenVINO Model Server, OVMS

This model can be served with [OpenVINO Model Server](https://github.com/openvinotoolkit/model_server) using the OpenAI-compatible `/v3/chat/completions` endpoint.

The workflow below was validated with:

- OpenVINO Model Server: `openvino/model_server:2026.1-gpu`
- OpenVINO backend: `2026.1`
- OpenVINO GenAI backend: `2026.1`
- Task: `text_generation`
- Device: `CPU`
- Endpoint: `/v3/chat/completions`

### 1. Clone the OpenVINO Model Server repository

The `model_server` folder used below comes from the official OpenVINO Model Server GitHub repository.

```bash
cd ~/ov_models

git clone https://github.com/openvinotoolkit/model_server.git
cd model_server

Create and activate a Python environment for the OVMS export helper:

python3 -m venv ovms_client_env
source ovms_client_env/bin/activate

python -m pip install --upgrade pip
pip install -r demos/common/export_models/requirements.txt
pip install requests huggingface_hub optimum optimum-intel openvino openvino-genai openvino-tokenizers nncf

2. Download this model locally

Install or update the Hugging Face CLI:

pip install -U huggingface_hub

Download the model:

cd ~/ov_models

hf download Morteza89/qwen2-57b-a14b-int4-ov \
  --local-dir /home/devcloud/ov_models/ov_Qwen2_57B_A14B_int4

3. Generate an OVMS model repository

Set the model variables:

cd ~/ov_models

MODEL_NAME="qwen2-57b-a14b-int4"
MODEL_DIR="/home/devcloud/ov_models/ov_Qwen2_57B_A14B_int4"
REPO_DIR="/home/devcloud/ov_models/ovms_qwen2_57b_a14b_repo"

Generate the OVMS repository and graph.pbtxt using the OpenVINO Model Server export helper:

cd ~/ov_models/model_server
source ovms_client_env/bin/activate

rm -rf "$REPO_DIR"
mkdir -p "$REPO_DIR"

python demos/common/export_models/export_model.py text_generation \
  --source_model "$MODEL_DIR" \
  --model_name "$MODEL_NAME" \
  --config_file_path "$REPO_DIR/config.json" \
  --model_repository_path "$REPO_DIR" \
  --overwrite_models

Verify the generated configuration:

cat "$REPO_DIR/config.json"
grep -n "models_path" "$REPO_DIR/$MODEL_NAME/graph.pbtxt"

4. Run the model with OVMS

Stop any existing OVMS container using the same port:

docker rm -f ovms-local-moe 2>/dev/null || true

Start OpenVINO Model Server:

docker run \
  --name ovms-local-moe \
  -d \
  -p 8000:8000 \
  -v "$REPO_DIR:/workspace:ro" \
  -v "$MODEL_DIR:$MODEL_DIR:ro" \
  openvino/model_server:2026.1-gpu \
  --rest_port 8000 \
  --config_path /workspace/config.json

5. Check model status

Wait a few seconds and check OVMS status:

sleep 10

docker logs ovms-local-moe --tail 200
curl -s http://localhost:8000/v1/config | jq .

The model is ready when the status becomes:

{
  "qwen2-57b-a14b-int4": {
    "model_version_status": [
      {
        "version": "1",
        "state": "AVAILABLE",
        "status": {
          "error_code": "OK",
          "error_message": "OK"
        }
      }
    ]
  }
}

6. Run text generation

Use the OpenAI-compatible chat completions endpoint:

curl -s http://localhost:8000/v3/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2-57b-a14b-int4",
    "max_tokens": 128,
    "temperature": 0,
    "stream": false,
    "messages": [
      {
        "role": "user",
        "content": "Explain OpenVINO Model Server in one short paragraph. Do not show reasoning."
      }
    ]
  }' | jq .

8. Stop the server
docker rm -f ovms-local-moe