Hugging Face's logo

AXERA-TECH
/
Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K

Image-Text-to-Text
Transformers
English
Chinese
Qwen3.5
Qwen3.5-2B
VLM
GPTQ
Int4
Model card Files
xet
 Community

Qwen3.5-2B

This version of Qwen3.5-2B has been converted to run on the Axera NPU using w4a16 quantization.

Compatible with Pulsar2 version: 5.0

Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo :

Pulsar2 Link, How to Convert LLM from Huggingface to axmodel

AXera NPU HOST LLM Runtime

Support Platform

Image Process

Chips input size image num ttft(168 tokens) w4a16 CMM Flash
AX650 384*384 1 368 ms 13.4 tokens/sec 1.90GiB 2.86GiB

Video Process

Chips input size image num ttft(600 tokens) w4a16 CMM Flash
AX650 384*384 8 892 ms 13.4 tokens/sec 1.90GiB 2.86GiB

The DDR capacity refers to the CMM memory that needs to be consumed. Ensure that the CMM memory allocation on the development board is greater than this value.

How to use

安装 axllm

方式一：克隆仓库后执行安装脚本：

git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
cd ax-llm
./install.sh

方式二：一行命令安装（默认分支 axllm）：

curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash

方式三：下载Github Actions CI 导出的可执行程序（适合没有编译环境的用户）：

如果没有编译环境，请到： https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm 下载 最新 CI 导出的可执行程序（axllm），然后：

chmod +x axllm
sudo mv axllm /usr/bin/axllm

模型下载（Hugging Face）

先创建模型目录并进入，然后下载到该目录：

mkdir -p AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C128-P1152-CTX2047
cd AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C128-P1152-CTX2047
hf download AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C128-P1152-CTX2047 --local-dir .

# structure of the downloaded files
tree -L 3
`-- AXERA-TECH
    `-- Qwen3.5-2B-AX650-GPTQ-Int4-C128-P1152-CTX2047
        |-- qwen3_5_vision.axmodel
        |-- README.md
        |-- config.json
        |-- image.png
        |-- model.embed_tokens.weight.bfloat16.bin
        |-- post_config.json
        |-- qwen3_5_tokenizer.txt
        |-- qwen3_5_text_p128_l0_together.axmodel
        ...
        |-- qwen3_5_text_p128_l23_together.axmodel
        |-- qwen3_5_text_post.axmodel
        `-- vision_cache

3 directories, 39 files

Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board

运行（CLI）

图片理解 

root@ax650 ~/yongqiang/lhj/Qwen3_5.AXERA/ax-llm # axllm run Qwen3.5-2B-AX650-GPTQ-Int4-C128-P1152-CTX2047/
11:40:21.412 INF Init:218 | LLM init start
11:40:21.412 INF Init:226 | mixed attention enabled: full_attention_interval=4 ref_full_layer_idx=3
tokenizer_type = 3
 96% | ##############################   |  26 /  27 [3.78s<3.93s, 6.87 count/s] init post axmodel ok,remain_cmm(5589 MB)
11:40:25.195 INF Init:368 | max_token_len : 2047
11:40:25.195 INF Init:371 | kv_cache_size : 512, kv_cache_num: 2047
11:40:25.195 INF Init:374 | prefill_token_num : 128
11:40:25.195 INF Init:379 | grp: 1, prefill_max_kv_cache_num : 1
11:40:25.195 INF Init:379 | grp: 2, prefill_max_kv_cache_num : 128
11:40:25.195 INF Init:379 | grp: 3, prefill_max_kv_cache_num : 256
11:40:25.195 INF Init:379 | grp: 4, prefill_max_kv_cache_num : 384
11:40:25.195 INF Init:379 | grp: 5, prefill_max_kv_cache_num : 512
11:40:25.195 INF Init:379 | grp: 6, prefill_max_kv_cache_num : 768
11:40:25.195 INF Init:379 | grp: 7, prefill_max_kv_cache_num : 1024
11:40:25.195 INF Init:379 | grp: 8, prefill_max_kv_cache_num : 1152
11:40:25.195 INF Init:384 | prefill_max_token_num : 1152
11:40:25.195 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  27 /  27 [3.79s<3.79s, 7.13 count/s] embed_selector init ok
11:40:25.478 INF Init:643 | Qwen-VL token ids: vision_start=248053 image_pad=248056 video_pad=248057
11:40:25.478 INF Init:668 | VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
11:40:25.478 WRN Init:677 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
11:40:25.480 INF load_config:282 | load config: 
11:40:25.480 INF load_config:282 | {
11:40:25.480 INF load_config:282 |     "enable_repetition_penalty": false,
11:40:25.480 INF load_config:282 |     "enable_temperature": false,
11:40:25.480 INF load_config:282 |     "enable_top_k_sampling": true,
11:40:25.480 INF load_config:282 |     "enable_top_p_sampling": false,
11:40:25.480 INF load_config:282 |     "penalty_window": 20,
11:40:25.480 INF load_config:282 |     "repetition_penalty": 1.2,
11:40:25.480 INF load_config:282 |     "temperature": 0.9,
11:40:25.480 INF load_config:282 |     "top_k": 10,
11:40:25.480 INF load_config:282 |     "top_p": 0.8
11:40:25.480 INF load_config:282 | }
11:40:25.480 INF Init:448 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
VLM enabled: after each prompt, input image path (empty = text-only). Use "video:<frames_dir>" for video.
----------------------------------------
prompt >> describe the image
image >> image.png
19:25:48.491 INF EncodeForContent:973 | Qwen-VL pixel_values[0] bytes=884736 min=0 max=238 (w=384 h=384 tp=2 ps=16 sm=2)
19:25:48.641 INF EncodeForContent:996 | vision cache store: image.png
19:25:48.665 INF SetKVCache:747 | prefill_grpid:3 kv_cache_num:256 precompute_len:0 input_num_token:168
19:25:48.665 INF SetKVCache:749 | current prefill_max_token_num:1152
19:25:48.665 INF SetKVCache:750 | first run
19:25:48.709 INF Run:805 | input token num : 168, prefill_split_num : 2
19:25:48.709 INF Run:845 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=128
19:25:48.710 INF Run:868 | prefill indices shape: p=0 idx_elems=128 idx_rows=1 pos_rows=3
19:25:48.881 INF Run:845 | prefill chunk p=1 history_len=128 grpid=2 kv_cache_num=128 input_tokens=40
19:25:48.881 INF Run:868 | prefill indices shape: p=1 idx_elems=128 idx_rows=1 pos_rows=3
19:25:49.077 INF Run:1010 | ttft: 368.31 ms
<think>
The user wants a description of the provided image.

1.  **Identify the main subjects:** There are three individuals wearing white astronaut-style spacesuits or heavy, space-inspired uniforms.
2.  **Describe their appearance:**
    *   They are dressed in bulky, white, bulky jackets with hoods.
    *   They are wearing helmets with metallic visors (the reflective visors are slightly reflective in the image).
    *   They appear to be standing amidst tall, leaf-like vegetation.
    *   The setting looks outdoors, possibly a forest or a jungle-like area, given the tall, grass-like plants.
3.  **Describe the atmosphere/mood**:
    *   It looks surreal or dreamlike.
    *   It doesn't clearly look like a real-world space station.
    *   The lighting is somewhat soft, and the colors are somewhat muted and desaturated, suggesting an artistic filter or a digital manipulation effect rather than a natural scene.
4.  **Specific details:
    - The vegetation:
    - It looks like a dense, grass-like forest.
</think>

This image features three figures wearing bulky, white, space-inspired suits and helmets with reflective visors. They stand among an abundance of tall, leaf-like vegetation, resembling a dense, grass-like forest. The atmosphere is surreal and dreamlike, the lighting and color scheme suggest a surreal or digitally manipulated style rather a realistic depiction of space stations.

19:26:11.722 NTC Run:1132 | hit eos,avg 13.42 token/s
19:26:11.722 INF GetKVCache:721 | precompute_len:340, remaining:812

视频理解 

root@ax650 ~/Qwen3.5-2B-AX650-GPTQ-Int4-C128-P1152-CTX2047 # ./axllm run ./
19:29:18.588 INF Init:218 | LLM init start
19:29:18.588 INF Init:226 | mixed attention enabled: full_attention_interval=4 ref_full_layer_idx=3
tokenizer_type = 3
 96% | ##############################   |  26 /  27 [17.06s<17.72s, 1.52 count/s] init post axmodel ok,remain_cmm(6157 MB)
19:29:35.651 INF Init:368 | max_token_len : 2047
19:29:35.651 INF Init:371 | kv_cache_size : 512, kv_cache_num: 2047
19:29:35.651 INF Init:374 | prefill_token_num : 128
19:29:35.651 INF Init:379 | grp: 1, prefill_max_kv_cache_num : 1
19:29:35.651 INF Init:379 | grp: 2, prefill_max_kv_cache_num : 128
19:29:35.651 INF Init:379 | grp: 3, prefill_max_kv_cache_num : 256
19:29:35.651 INF Init:379 | grp: 4, prefill_max_kv_cache_num : 384
19:29:35.651 INF Init:379 | grp: 5, prefill_max_kv_cache_num : 512
19:29:35.651 INF Init:379 | grp: 6, prefill_max_kv_cache_num : 640
19:29:35.651 INF Init:379 | grp: 7, prefill_max_kv_cache_num : 768
19:29:35.651 INF Init:379 | grp: 8, prefill_max_kv_cache_num : 896
19:29:35.651 INF Init:379 | grp: 9, prefill_max_kv_cache_num : 1024
19:29:35.651 INF Init:379 | grp: 10, prefill_max_kv_cache_num : 1152
19:29:35.651 INF Init:384 | prefill_max_token_num : 1152
19:29:35.651 INF Init:27 | LLaMaEmbedSelector use mmap
100% | ################################ |  27 /  27 [17.07s<17.07s, 1.58 count/s] embed_selector init ok
19:29:35.998 INF Init:643 | Qwen-VL token ids: vision_start=248053 image_pad=248056 video_pad=248057
19:29:35.998 INF Init:668 | VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
19:29:35.998 WRN Init:677 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
19:29:35.999 INF load_config:282 | load config: 
19:29:35.999 INF load_config:282 | {
19:29:35.999 INF load_config:282 |     "enable_repetition_penalty": false,
19:29:35.999 INF load_config:282 |     "enable_temperature": false,
19:29:35.999 INF load_config:282 |     "enable_top_k_sampling": true,
19:29:35.999 INF load_config:282 |     "enable_top_p_sampling": false,
19:29:35.999 INF load_config:282 |     "penalty_window": 20,
19:29:35.999 INF load_config:282 |     "repetition_penalty": 1.2,
19:29:35.999 INF load_config:282 |     "temperature": 0.9,
19:29:35.999 INF load_config:282 |     "top_k": 10,
19:29:35.999 INF load_config:282 |     "top_p": 0.8
19:29:35.999 INF load_config:282 | }
19:29:36.000 INF Init:448 | LLM init ok
Commands:
  /q, /exit  退出
  /reset     重置 kvcache
  /dd        删除一轮对话
  /pp        打印历史对话
Ctrl+C: 停止当前生成
VLM enabled: after each prompt, input image path (empty = text-only). Use "video:<frames_dir>" for video.
----------------------------------------
prompt >> 描述视频内容
media >> video:assets/football.mp4:1
20:52:14.133 INF extract_video_frames_ffmpeg:299 | Extracting raw video container to frames: assets/football.mp4 -> /tmp/axllm_video_frames/video_87979794180908_0 fps=0.99900099900099892
20:52:33.025 INF collect_video_frame_paths:379 | Video fps sampling: path=assets/football.mp4 fps=1 duration=60.060s target_frames=60 selected=60
20:52:46.835 INF SetKVCache:2543 | decode_grpid:2 prefill_grpid:4 history_cap:0 total_cap:256 symbolic_cap:1 precompute_len:0 input_num_token:4344 prefer_symbolic_group:0
20:52:46.835 INF SetKVCache:2565 | current prefill_max_token_num:6400
20:52:46.936 INF SetKVCache:2581 | first run
20:52:46.992 INF Run:2738 | input token num : 4344, prefill_split_num : 17
20:52:46.992 INF Run:2818 | prefill chunk p=0 history_len=0 grpid=4 kv_cache_num=0 input_tokens=256
20:52:47.327 INF Run:2818 | prefill chunk p=1 history_len=256 grpid=6 kv_cache_num=512 input_tokens=256
20:52:47.689 INF Run:2818 | prefill chunk p=2 history_len=512 grpid=7 kv_cache_num=768 input_tokens=256
20:52:48.059 INF Run:2818 | prefill chunk p=3 history_len=768 grpid=8 kv_cache_num=1024 input_tokens=256
20:52:48.436 INF Run:2818 | prefill chunk p=4 history_len=1024 grpid=9 kv_cache_num=1280 input_tokens=256
20:52:48.820 INF Run:2818 | prefill chunk p=5 history_len=1280 grpid=10 kv_cache_num=1536 input_tokens=256
20:52:49.216 INF Run:2818 | prefill chunk p=6 history_len=1536 grpid=11 kv_cache_num=1792 input_tokens=256
20:52:49.625 INF Run:2818 | prefill chunk p=7 history_len=1792 grpid=12 kv_cache_num=2048 input_tokens=256
20:52:50.049 INF Run:2818 | prefill chunk p=8 history_len=2048 grpid=13 kv_cache_num=2304 input_tokens=256
20:52:50.468 INF Run:2818 | prefill chunk p=9 history_len=2304 grpid=14 kv_cache_num=2560 input_tokens=256
20:52:50.893 INF Run:2818 | prefill chunk p=10 history_len=2560 grpid=15 kv_cache_num=2816 input_tokens=256
20:52:51.321 INF Run:2818 | prefill chunk p=11 history_len=2816 grpid=16 kv_cache_num=3072 input_tokens=256
20:52:51.751 INF Run:2818 | prefill chunk p=12 history_len=3072 grpid=17 kv_cache_num=3328 input_tokens=256
20:52:52.180 INF Run:2818 | prefill chunk p=13 history_len=3328 grpid=18 kv_cache_num=3584 input_tokens=256
20:52:52.619 INF Run:2818 | prefill chunk p=14 history_len=3584 grpid=19 kv_cache_num=3840 input_tokens=256
20:52:53.066 INF Run:2818 | prefill chunk p=15 history_len=3840 grpid=20 kv_cache_num=4096 input_tokens=256
20:52:53.519 INF Run:2818 | prefill chunk p=16 history_len=4096 grpid=21 kv_cache_num=4352 input_tokens=248
20:52:54.031 INF Run:3045 | ttft: 7038.25 ms
<think>20:52:54.031 INF Run:3076 | VLM decode positions: rope_start=4212 dense_kv_start=4344


</think>

好的，这张视频截图展示了一个足球比赛的精彩瞬间，并配有解说员的反应。以下是详细描述：

- **主要画面**：
    - 视频的主体部分是一个足球比赛的实时画面。
    - 比赛场地是绿色的草坪，画有标准的白色界线。
    - 场上有两名球队：一支穿着白色球衣的球队和一支穿着紫红色条纹球衣的球队。
    - 画面捕捉到了一个进攻场景，一名身穿紫红色球衣的球员（背号9号，根据后续画面可知是苏亚雷斯）在禁区附近接到传球后，起脚射门。
    - 球被守门员（身穿橙色球衣）扑出，但球已经飞入球门。
    - 进球后，身穿紫红色球衣的球员兴奋地冲向球门庆祝。

- **画中画（PPT）区域**：
    - 在右上角，有一个较小的视频窗口，显示了一位男性解说员。
    - 他留着黑色短发，戴着黑色针织帽，身穿黄黑相间的条纹运动衫。
    - 他正看着屏幕上的进球，表情激动，双手做出庆祝和鼓掌的动作，似乎在为进球欢呼。
    - 他的背景墙上挂着一些奖杯和照片。

- **文字信息**：
    - 视频左上角有文字：“Barcelona vs Sevilla 6-1 All Goals & Extended Highlights 30/01/2019 HD”。这表明这是巴塞罗那队与塞维利亚队的一场足球比赛，比分是6-1，并且包含了所有进球和延长时间的精彩集锦。
    - 视频左下角有“FC2019MPS10”的水印。

20:53:24.838 NTC Run:3445 | hit eos,decode avg 11.43 token/s
20:53:24.838 INF GetKVCache:2496 | precompute_len:4697, remaining:1703 (tracked)

启动服务（OpenAI 兼容） 

root@ax650:~# axllm serve AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C128-P1152-CTX2047
[I][                            Init][ 138]: LLM init start
tokenizer_type = 1
 96% | ███████████████████████████████   |  30 /  31 [4.63s<4.79s, 6.47 count/s] init post axmodel ok,remain_cmm(9563 MB)
[I][                            Init][ 199]: max_token_len : 2047
[I][                            Init][ 202]: kv_cache_size : 1024, kv_cache_num: 2047
[I][                            Init][ 205]: prefill_token_num : 128
[I][                            Init][ 209]: grp: 1, prefill_max_kv_cache_num : 1
[I][                            Init][ 209]: grp: 2, prefill_max_kv_cache_num : 128
[I][                            Init][ 209]: grp: 3, prefill_max_kv_cache_num : 256
[I][                            Init][ 209]: grp: 4, prefill_max_kv_cache_num : 384
[I][                            Init][ 209]: grp: 5, prefill_max_kv_cache_num : 512
[I][                            Init][ 209]: grp: 6, prefill_max_kv_cache_num : 640
[I][                            Init][ 209]: grp: 7, prefill_max_kv_cache_num : 768
[I][                            Init][ 209]: grp: 8, prefill_max_kv_cache_num : 896
[I][                            Init][ 209]: grp: 9, prefill_max_kv_cache_num : 1024
[I][                            Init][ 209]: grp: 10, prefill_max_kv_cache_num : 1152
[I][                            Init][ 214]: prefill_max_token_num : 1152
[I][                            Init][  27]: LLaMaEmbedSelector use mmap
100% | ████████████████████████████████ |  31 /  31 [4.64s<4.64s, 6.69 count/s] embed_selector init ok
[W][                            Init][ 457]: Qwen-VL vision size override: cfg=448x448 bytes=1204224, model_input_bytes=884736 -> 384x384 (square).
[I][                            Init][ 641]: Qwen-VL token ids: vision_start=151652 image_pad=151655 video_pad=151656
[I][                            Init][ 666]: VisionModule init ok: type=Qwen3VL, tokens_per_block=144, embed_size=2048, out_dtype=fp32
[I][                            Init][ 672]: VisionModule deepstack enabled: layers=3
[I][                     load_config][ 282]: load config:
{
    "enable_repetition_penalty": false,
    "enable_temperature": false,
    "enable_top_k_sampling": false,
    "enable_top_p_sampling": false,
    "penalty_window": 20,
    "repetition_penalty": 1.2,
    "temperature": 0.9,
    "top_k": 10,
    "top_p": 0.8
}

[I][                            Init][ 272]: LLM init ok
Starting server on port 8000 with model 'AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C128-P1152-CTX2047'...
OpenAI API Server starting on http://0.0.0.0:8000
Max concurrency: 1
Models: AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C128-P1152-CTX2047

OpenAI 调用示例 

from openai import OpenAI

API_URL = "http://127.0.0.1:8000/v1"
MODEL = "AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C128-P1152-CTX2047"

messages = [
    {"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]},
    {"role": "user", "content": "hello"},
]

client = OpenAI(api_key="not-needed", base_url=API_URL)
completion = client.chat.completions.create(
    model=MODEL,
    messages=messages,
)

print(completion.choices[0].message.content)

OpenAI 流式调用示例 

from openai import OpenAI

API_URL = "http://127.0.0.1:8000/v1"
MODEL = "AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C128-P1152-CTX2047"

messages = [
    {"role": "system", "content": [{"type": "text", "text": "you are a helpful assistant."}]},
    {"role": "user", "content": "hello"},
]

client = OpenAI(api_key="not-needed", base_url=API_URL)
stream = client.chat.completions.create(
    model=MODEL,
    messages=messages,
    stream=True,
)

print("assistant:")
for ev in stream:
    delta = getattr(ev.choices[0], "delta", None)
    if delta and getattr(delta, "content", None):
        print(delta.content, end="", flush=True)
print("
")
Downloads last month
15
Inference Providers NEW
Image-Text-to-Text
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K

Base model

Qwen/Qwen3.5-2B-Base
Finetuned
Qwen/Qwen3.5-2B
Finetuned
(254)
this model

Collection including AXERA-TECH/Qwen3.5-2B-AX650-GPTQ-Int4-C256-P6K-CTX8K