---
library_name: transformers
license: apache-2.0
base_model:
  - openbmb/MiniCPM-V-4.6-GPTQ
pipeline_tag: image-text-to-text
tags:
  - minicpm-v
  - vlm
  - gptq
  - axera
  - ax650
language:
  - zh
  - en
---

# MiniCPM-V-4.6-GPTQ on AXERA NPU

Ready-to-run deployment package for `openbmb/MiniCPM-V-4.6-GPTQ` on AX650 / NPU3.

- This release packages the AX650 `axllm` runtime together with the compiled text and vision `.axmodel` files.
- The packaged text runtime uses the GPTQ INT4 build.
- The packaged vision runtime uses a fixed-shape `448x448` MiniCPM-V-4.6 vision encoder.
- The package supports text-only chat, single-image understanding, and video understanding through the OpenAI-compatible `axllm serve` API.
- The package includes sample assets for image and video validation.

## Supported Platform

- [x] AX650 / NPU3

## Validated Devices

This package has been validated on the following AX650-based device:

- AX650 / NPU3 development board

## Performance

All measurements below were taken on AX650 / NPU3. `TTFT` stands for time to first token. In this table, `TTFT` is measured end-to-end from request arrival at `axllm serve` to the first generated token, so the multimodal rows include media preprocessing and vision encoding time.

The text-only smoke prompt was kept within one `128`-token prefill chunk. To avoid one-time startup effects, the text row below excludes the first request after service startup. Its `Decode` figure was measured with longer text-only generations (`max_tokens=256`) to better reflect sustained decode throughput; the short smoke reply used for the `TTFT` row is effectively a single-token answer and would otherwise under-report decode speed. The image row was measured with the packaged fixed-shape `448x448` vision encoder and `assets/sample.png`. The video row used the packaged sample video with `video:assets/red-panda-openai.mp4:2`.

| Scenario | Input tokens | Prefill chunks | TTFT | Decode |
|---|---:|---:|---:|---:|
| Text-only smoke prompt | `25` | `1 x 128` | `260.81 ms avg` (`259.01-262.61 ms`) | `24.07 token/s avg` |
| Image prompt | `88` | `1 x 128` | `719.79 ms avg` (`708.57-732.47 ms`) | `24.49 token/s avg` |
| Video prompt | `1271` | `10 x 128` | `9555.33 ms avg` (`9484.00-9647.62 ms`) | `23.87 token/s avg` |

The packaged runtime uses the following context layout:

- `prefill_len=128`
- `kv_cache_len=2047`
- `prefill_max_token_num=1280`

`Input tokens` in the table above refers to the full request length after chat templating, not just the visual soft tokens. For the shipped `448x448` vision encoder, each selected image block contributes `64` visual soft tokens. Under the current packaged runtime settings, the sample video request in this README uses `1271` total input tokens and spans `10` prefill chunks.

### Startup Runtime Footprint

| Item | Value |
|---|---:|
| `Flash total (text + post + vision axmodels)` | `1.19 GiB` (`1214.38 MiB`) |
| `Package flash total (current repository layout, excluding runtime-generated vision_cache/)` | `1.68 GiB` (`1719.30 MiB`) |
| `Runtime CMM increment during board-side startup` | `1.30 GiB` (`1334.05 MiB`) |

The runtime CMM value above was measured during board-side startup on a shared AX650 system and should be treated as a practical reference value.

## Vision Encoder Latency

Measured on AX650 / NPU3 with `/opt/bin/ax_run_model -m minicpmv4_6_vision_448.axmodel -g 0 -w 1 -r 5`.

| Model | Resolution | Soft Tokens | Time (ms) |
|---|---|---|---:|
| `minicpmv4_6_vision_448.axmodel` | `448x448` | `64` | `235.285 ms avg` |

For this packaged AX650 runtime, the visual token count is fixed by the shipped vision encoder configuration:

- `vision_width = 448`
- `vision_height = 448`
- `vision_patch_size = 14`
- patch grid = `(448 / 14) x (448 / 14) = 32 x 32`
- raw patch tokens = `32 x 32 = 1024`
- current packaged build uses the `16x` visual compression path
- `Soft Tokens = 1024 / 16 = 64`

So, for the fixed-shape runtime shipped in this repository, the relation is:

```text
Soft Tokens = (vision_width / patch_size) x (vision_height / patch_size) / 16
```

`Input tokens` in the performance table can be larger than the visual `Soft Tokens` because `axllm` counts the full templated request, including user text and chat-template tokens in addition to the visual tokens. For the packaged `assets/sample.png` request in this README, the runtime reports `input_num_token=88`, which still fits within a single `128`-token prefill chunk.

`Soft Tokens` is not a runtime-configurable value in this package. This repository ships only `minicpmv4_6_vision_448.axmodel`, so the board-side AX650 runtime always uses `448x448 -> 64` soft tokens for image encoding.

## Package Layout

```text
.
├── README.md
├── bin/
│   ├── axllm
│   └── axllm.version.json
├── assets/
│   ├── openai_api_demo.png
│   ├── red-panda-openai.mp4
│   └── sample.png
├── minicpmv4_6_vision_448.axmodel
├── qwen3_5_text_p128_l0_together.axmodel
├── ...
├── qwen3_5_text_p128_l23_together.axmodel
├── qwen3_5_text_post.axmodel
├── model.embed_tokens.weight.bfloat16.bin
├── config.json
├── post_config.json
└── minicpm_v46_tokenizer.txt
```

This package keeps the runtime files at the repository root so it can be served directly by `axllm`.

## Sample Image

Both the `axllm` flow and the packaged sample requests can use the sample image:
`assets/sample.png`

![sample](assets/sample.png)

## Sample Video

The package also includes a packaged sample video for board-side video understanding validation:

- `assets/red-panda-openai.mp4`

## Direct Inference with `axllm`

> The `axllm` workflow is still being refined. The instructions below reflect the current validated flow and may be adjusted as the packaging continues to evolve.

### Download the Model Package

Download the release package from Hugging Face:

```shell
mkdir -p AXERA-TECH/MiniCPM-V-4.6-GPTQ
cd AXERA-TECH/MiniCPM-V-4.6-GPTQ
hf download AXERA-TECH/MiniCPM-V-4.6-GPTQ --local-dir .
```

### Install `axllm`

Option 1: use the validated binary included in this repository:

```bash
chmod +x ./bin/axllm
```

Option 2: install `axllm` from the public repository:

```shell
git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh
```

Option 3: install with a one-line command:

```shell
curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash
```

Option 4: download the prebuilt binary from GitHub Actions CI:

If you do not have a local build environment, download the latest CI-generated `axllm` binary from GitHub Actions:
`https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm`
Then run:

```shell
chmod +x axllm
sudo mv axllm /usr/bin/axllm
```

### Run on the Board

The package root is already arranged for `axllm`, so no extra runtime path arguments are required.

For multimodal testing, you can use the packaged sample image shown above: `./assets/sample.png`, or the packaged sample video: `./assets/red-panda-openai.mp4`.

```bash
./bin/axllm run .
```

In interactive mode:

- press `Enter` directly for text-only chat
- input an image path for single-image chat
- input `video:/path/to/frames_dir` or `video:/path/to/video.mp4` for video chat

### Serve with `axllm`

From the package root on the board:

```bash
./bin/axllm serve . --port 8000
```

Expected model id:

```text
AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-C128-P1152-CTX2047
```

Health check:

```bash
curl http://127.0.0.1:8000/health
```

A typical startup log looks like this:

```text
INF Init | LLM init start
INF Init | mixed attention enabled: full_attention_interval=4 ref_full_layer_idx=3
INF Init | attention config: layers=24 sliding=0 full=6 linear=18 sliding_window=0 ref_full_layer_idx=3
tokenizer_type = 3
huggingface tokenizer mode = gpt2_byte_bpe
...
INF Init | max_token_len : 2047
INF Init | kv_cache_size : 512, kv_cache_num: 2047
INF init_groups_from_model | prefill_token_num : 128
INF init_groups_from_model | prefill_max_token_num : 1280
INF Init | MiniCPM-V-4.6 token ids: image_pad=248056 video_pad=248057
INF Init | VisionModule init ok: type=MiniCPMV46VL, tokens_per_block=64, embed_size=1024, out_dtype=fp32
INF Init | LLM init ok
Starting server on port 8000 with model 'AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-C128-P1152-CTX2047'...
API URLs:
  GET  http://127.0.0.1:8000/health
  GET  http://127.0.0.1:8000/v1/models
  POST http://127.0.0.1:8000/v1/chat/completions
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-C128-P1152-CTX2047
```

You can then send requests to the server using the API endpoints shown in the log. For example, to check the health status and list the available models:

```bash
curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/v1/models
```

Example output:

```json
{
  "concurrency": 0,
  "max_concurrency": 1,
  "status": "healthy"
}
{
  "data": [
    {
      "created": 1780911663,
      "id": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-C128-P1152-CTX2047",
      "object": "model",
      "owned_by": "openai-api"
    }
  ],
  "object": "list"
}
```

![openai_api_demo](assets/openai_api_demo.png)

### Text Request

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-C128-P1152-CTX2047",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "What is 1+1? Reply with the number only."}
        ]
      }
    ],
    "max_tokens": 32
  }'
```

Example output:

```json
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "2"
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-C128-P1152-CTX2047",
  "object": "chat.completion"
}
```

### Image Request

```bash
python3 - <<'PY'
import base64
import json
from pathlib import Path
from urllib.request import Request, urlopen

img = Path("assets/sample.png").read_bytes()
payload = {
    "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-C128-P1152-CTX2047",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Please briefly describe this image."},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/png;base64," + base64.b64encode(img).decode()
                    },
                },
            ],
        }
    ],
    "max_tokens": 64,
}
req = Request(
    "http://127.0.0.1:8000/v1/chat/completions",
    data=json.dumps(payload).encode(),
    headers={"Content-Type": "application/json"},
)
with urlopen(req, timeout=60) as resp:
    print(resp.read().decode())
PY
```

Example output:

```json
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "The image shows a colorful, cartoon-style red lobster or lobster-like character with a cheerful expression, raised claws, and a dynamic, action-oriented pose."
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-C128-P1152-CTX2047",
  "object": "chat.completion"
}
```

### Video Request

`axllm serve` accepts either a frames directory or a raw video file:

```bash
curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-C128-P1152-CTX2047",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "video:/path/to/frames_dir"}},
          {"type": "text", "text": "Describe this video briefly."}
        ]
      }
    ],
    "max_tokens": 128
  }'
```

For a raw video file, use `video:/path/to/video.mp4`. If you need to request a specific sampling FPS, use the form `video:/path/to/video.mp4:2`.

To test the packaged sample video from the package root, you can set:

```bash
VIDEO_PATH="$(pwd)/assets/red-panda-openai.mp4"
```

and then use `video:${VIDEO_PATH}:2` in the request payload.

Example output:

```json
{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "The red panda is seen playing with the other red panda."
      },
      "finish_reason": "stop"
    }
  ],
  "model": "AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-C128-P1152-CTX2047",
  "object": "chat.completion"
}
```

### Browser UI with `lite_webui`

If you want a browser UI for the OpenAI-compatible service started by `axllm serve`, use [AXERA-TECH/lite_webui](https://huggingface.co/AXERA-TECH/lite_webui/tree/main).

Set the OpenAI base URL to `http://<board-ip>:8000` and the model name to `AXERA-TECH/MiniCPM-V-4.6-GPTQ-AX650-C128-P1152-CTX2047`.

## Conversion References

If you need the original model files or want to rebuild the deployment artifacts, start with:

- Original Hugging Face model: [openbmb/MiniCPM-V-4.6-GPTQ](https://huggingface.co/openbmb/MiniCPM-V-4.6-GPTQ)
- AXERA conversion and deployment workflow: [AXERA-TECH/MiniCPM-V-4.6.axera](https://github.com/AXERA-TECH/MiniCPM-V-4.6.axera)

## Discussion

- GitHub Issues
- QQ group: `139953715`