Align gemma4 runtime layout and refresh Python deployment docs

.gitattributes

@@ -42,3 +42,4 @@ main_axcl_x86 filter=lfs diff=lfs merge=lfs -text
 *.jpg filter=lfs diff=lfs merge=lfs -text
 *.mp4 filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+gemma4_tokenizer.txt filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -7,3 +7,4 @@ dist/
 .codex
 .claude
 .gemini
+*cache*

README.md

@@ -13,7 +13,7 @@ language:
 ---
 
 <p align="center">
-  <img src="assets/gemma4_axera_banner.png" alt="Gemma4-Axera Banner">
+  <img src="assets/gemma4_axera_banner.jpg" alt="Gemma4-Axera Banner">
 </p>
 
 # Gemma 4 E2B on AXERA NPU
@@ -26,13 +26,6 @@ Ready-to-run deployment package for `google/gemma-4-E2B-it` on AX650 / NPU3.
 - Includes compiled Gemma 4 text `.axmodel` files and Vision `.axmodel` files.
 - Supports both text-only chat and single-image multimodal inference.
 
-## Conversion References
-
-If you need the original model files or want to rebuild the deployment artifacts, start with:
-
-- Original Hugging Face model: [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it)
-- AXERA conversion and deployment workflow: [AXERA-TECH/gemma-4-E2B-it.axera](https://github.com/AXERA-TECH/gemma-4-E2B-it.axera)
-
 ## Supported Platform
 
 - [x] AX650 / NPU3
@@ -52,7 +45,7 @@ All measurements below were taken on AX650 / NPU3. `TTFT` stands for time to fir
 - `w8a16`: TTFT is approximately `1664 ms`, with a decode throughput of approximately `10.44 tokens/s`.
 - `w4a16`: TTFT is approximately `1233.7 ms`, with a decode throughput of approximately `15.22 tokens/s`.
 
-The packaged text runtime in this release is the `w8a16` build located in `gemma_4_e2b_it_ax650n_axmodel`. The `w4a16` numbers are provided for reference only.
+The packaged text runtime in this release is the `w8a16` build. Its text runtime files are packaged at the repository root. The `w4a16` numbers are provided for reference only.
 
 ## Vision Encoder Latency
 
@@ -68,18 +61,275 @@ The packaged text runtime in this release is the `w8a16` build located in `gemma
 .
 ├── README.md
 ├── config.json
+├── post_config.json
 ├── infer_axmodel.py
 ├── gradio_demo.py
 ├── assets/
 ├── gemma_4_e2b_it_tokenizer/
-├── gemma_4_e2b_it_ax650n_axmodel/
+├── gemma4_tokenizer.txt
+├── gemma4_text_p128_l*.axmodel
+├── gemma4_text_post.axmodel
+├── gemma4_vision_h336_w480_t70.axmodel
+├── gemma4_vision_h480_w672_t140.axmodel
+├── gemma4_vision_h672_w960_t280.axmodel
+├── model.embed_tokens_per_layer.weight.npy
+├── model.embed_tokens.weight.bfloat16.bin
+├── model.per_layer_model_projection.weight.npy
+├── model.per_layer_projection_norm.weight.npy
 ├── vit_models/
 └── utils/
 ```
 
-The runtime scripts auto-detect the packaged directories above. If you keep this layout unchanged, you can run the examples below without passing extra path arguments.
+This package uses a hybrid layout: the tokenizer stays in a subdirectory, the packaged text runtime files and Vision `.axmodel` files live at the repository root, and `vit_models/` keeps the accompanying Vision metadata JSON files.
+
+The Python demo scripts auto-detect the packaged paths above. If you keep this layout unchanged, you can run the Python examples later in this README without passing extra path arguments.
+
+## Sample Image
+
+Both the `axllm` flow and the legacy Python demo flow below can use the packaged sample image:
+`assets/sample.png`
+
+![sample](assets/sample.png)
+
+## Direct Inference with `axllm`
+
+> The `axllm` workflow is still being refined. The instructions below reflect the current validated flow and may be adjusted as the packaging continues to evolve.
+
+### Download the Model Package
+
+Download the release package from Hugging Face:
+
+```shell
+mkdir -p AXERA-TECH/gemma-4-E2B-it
+cd AXERA-TECH/gemma-4-E2B-it
+hf download AXERA-TECH/gemma-4-E2B-it --local-dir .
+```
+
+### Install `axllm`
+
+Option 1: clone the repository and run the installer:
+
+```shell
+git clone -b axllm https://github.com/AXERA-TECH/ax-llm.git
+cd ax-llm
+./install.sh
+```
+
+Option 2: install with a one-line command (default branch: `axllm`):
+
+```shell
+curl -fsSL https://raw.githubusercontent.com/AXERA-TECH/ax-llm/axllm/install.sh | bash
+```
+
+Option 3: download the prebuilt binary from GitHub Actions CI:
+
+If you do not have a local build environment, download the latest CI-generated `axllm` binary from GitHub Actions:
+`https://github.com/AXERA-TECH/ax-llm/actions?query=branch%3Aaxllm`
+Then run:
+
+```shell
+chmod +x axllm
+sudo mv axllm /usr/bin/axllm
+```
 
-## Runtime Requirements
+### Run on the Board
+
+The package root is already arranged for `axllm`, so no extra runtime path arguments are required.
+
+Note: the command below assumes you run it from the parent directory of `AXERA-TECH/gemma-4-E2B-it`. If you are already inside the package directory, use `axllm run .` instead.
+
+For multimodal testing, you can use the sample image shown above: `./assets/sample.png`.
+
+```bash
+$ axllm run AXERA-TECH/gemma-4-E2B-it
+
+# output log example:
+15:04:24.522 INF Init:890 | LLM init start
+15:04:24.522 INF Init:905 | shared kv enabled: num_kv_shared_layers=20
+tokenizer_type = 3
+huggingface tokenizer mode = space_replace_bpe
+ 31% | ##########                       |  12 /  38 [4.47s<14.16s, 2.68 count/s] init 10 axmodel ok,remain_cmm(6047 MB                                                                                                                       34% | ##########                       |  13 /  38 [4.61s<13.48s, 2.82 count/s] init 11 axmodel ok,remain_cmm(5992 MB                                                                                                                       36% | ###########                      |  14 /  38 [4.78s<12.98s, 2.93 count/s] init 12 axmodel ok,remain_cmm(5937 MB                                                                                                                       39% | ############                     |  15 /  38 [4.93s<12.49s, 3.04 count/s] init 13 axmodel ok,remain_cmm(5882 MB                                                                                                                       42% | #############                    |  16 /  38 [5.09s<12.09s, 3.14 count/s] init 14 axmodel ok,remain_cmm(5813 MB                                                                                                                       44% | ##############                   |  17 /  38 [5.28s<11.80s, 3.22 count/s] init 15 axmodel ok,remain_cmm(5727 MB                                                                                                                       47% | ###############                  |  18 /  38 [5.50s<11.61s, 3.27 count/s] init 16 axmodel ok,remain_cmm(5642 MB                                                                                                                       50% | ################                 |  19 /  38 [5.69s<11.38s, 3.34 count/s] init 17 axmodel ok,remain_cmm(5557 MB                                                                                                                       52% | ################                 |  20 /  38 [5.91s<11.22s, 3.39 count/s] init 18 axmodel ok,remain_cmm(5471 MB                                                                                                                       55% | #################                |  21 /  38 [6.11s<11.06s, 3.44 count/s] init 19 axmodel ok,remain_cmm(5373 MB                                                                                                                       57% | ##################               |  22 /  38 [6.31s<10.89s, 3.49 count/s] init 20 axmodel ok,remain_cmm(5287 MB                                                                                                                       60% | ###################              |  23 /  38 [6.53s<10.79s, 3.52 count/s] init 21 axmodel ok,remain_cmm(5202 MB                                                                                                                       63% | ####################             |  24 /  38 [6.75s<10.69s, 3.56 count/s] init 22 axmodel ok,remain_cmm(5117 MB                                                                                                                       65% | #####################            |  25 /  38 [6.96s<10.58s, 3.59 count/s] init 23 axmodel ok,remain_cmm(5031 MB                                                                                                                       68% | #####################            |  26 /  38 [7.18s<10.50s, 3.62 count/s] init 24 axmodel ok,remain_cmm(4933 MB                                                                                                                       71% | ######################           |  27 /  38 [7.40s<10.41s, 3.65 count/s] init 25 axmodel ok,remain_cmm(4847 MB                                                                                                                       73% | #######################          |  28 /  38 [7.62s<10.34s, 3.67 count/s] init 26 axmodel ok,remain_cmm(4762 MB                                                                                                                       76% | ########################         |  29 /  38 [7.85s<10.28s, 3.70 count/s] init 27 axmodel ok,remain_cmm(4676 MB                                                                                                                       78% | #########################        |  30 /  38 [8.13s<10.29s, 3.69 count/s] init 28 axmodel ok,remain_cmm(4591 MB                                                                                                                       81% | ##########################       |  31 /  38 [8.36s<10.25s, 3.71 count/s] init 29 axmodel ok,remain_cmm(4492 MB                                                                                                                       84% | ##########################       |  32 /  38 [8.60s<10.21s, 3.72 count/s] init 30 axmodel ok,remain_cmm(4407 MB                                                                                                                       86% | ###########################      |  33 /  38 [8.86s<10.21s, 3.72 count/s] init 31 axmodel ok,remain_cmm(4322 MB                                                                                                                       89% | ############################     |  34 /  38 [9.11s<10.18s, 3.73 count/s] init 32 axmodel ok,remain_cmm(4236 MB                                                                                                                       92% | #############################    |  35 /  38 [9.36s<10.16s, 3.74 count/s] init 33 axmodel ok,remain_cmm(4151 MB                                                                                                                       94% | ##############################   |  36 /  38 [9.62s<10.16s, 3.74 count/s] init 34 axmodel ok,remain_cmm(4052 MB                                                                                                                       97% | ###############################  |  37 /  38 [10.03s<10.30s, 3.69 count/s] init post axmodel ok,remain_cmm(3632 MB)
+15:04:34.551 INF Init:1045 | max_token_len : 2047
+15:04:34.551 INF Init:1048 | kv_cache_size : 256, kv_cache_num: 2047
+15:04:34.551 INF init_groups_from_model:606 | prefill_token_num : 128
+15:04:34.551 INF init_groups_from_model:820 | decode grp: 0, gid: 0, max_token_len : 2047
+15:04:34.551 INF init_groups_from_model:824 | prefill grp: 0, gid: 1, history_cap: 0, total_cap: 128, symbolic_cap: 1
+15:04:34.551 INF init_groups_from_model:824 | prefill grp: 1, gid: 2, history_cap: 128, total_cap: 256, symbolic_cap: 128
+15:04:34.551 INF init_groups_from_model:824 | prefill grp: 2, gid: 3, history_cap: 256, total_cap: 384, symbolic_cap: 256
+15:04:34.551 INF init_groups_from_model:824 | prefill grp: 3, gid: 4, history_cap: 384, total_cap: 512, symbolic_cap: 384
+15:04:34.551 INF init_groups_from_model:824 | prefill grp: 4, gid: 5, history_cap: 512, total_cap: 640, symbolic_cap: 512
+15:04:34.551 INF init_groups_from_model:824 | prefill grp: 5, gid: 6, history_cap: 640, total_cap: 768, symbolic_cap: 640
+15:04:34.551 INF init_groups_from_model:824 | prefill grp: 6, gid: 7, history_cap: 768, total_cap: 896, symbolic_cap: 768
+15:04:34.551 INF init_groups_from_model:824 | prefill grp: 7, gid: 8, history_cap: 896, total_cap: 1024, symbolic_cap: 896
+15:04:34.551 INF init_groups_from_model:824 | prefill grp: 8, gid: 9, history_cap: 1024, total_cap: 1152, symbolic_cap: 1024
+15:04:34.551 INF init_groups_from_model:831 | prefill_max_token_num : 1152
+15:04:34.551 INF Init:27 | LLaMaEmbedSelector use mmap
+100% | ################################ |  38 /  38 [10.03s<10.03s, 3.79 count/s] embed_selector init ok
+15:04:34.567 INF Init:475 | Gemma4 per-layer helper enabled: vocab=262144 hidden=1536 layers=35 per_layer=256 pad=0
+15:04:34.727 INF Init:785 | Gemma4-VL token ids: image_pad=258880 video_pad=258884
+15:04:34.727 INF Init:792 | VisionModule init ok: type=Gemma4VL, tokens_per_block=70, embed_size=1536, out_dtype=fp32
+15:04:34.727 WRN Init:801 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
+15:04:34.729 INF load_config:282 | load config:
+15:04:34.729 INF load_config:282 | {
+15:04:34.729 INF load_config:282 |     "enable_repetition_penalty": false,
+15:04:34.729 INF load_config:282 |     "enable_temperature": true,
+15:04:34.729 INF load_config:282 |     "enable_top_k_sampling": false,
+15:04:34.729 INF load_config:282 |     "enable_top_p_sampling": true,
+15:04:34.729 INF load_config:282 |     "penalty_window": 64,
+15:04:34.729 INF load_config:282 |     "repetition_penalty": 1.0,
+15:04:34.729 INF load_config:282 |     "temperature": 1.0,
+15:04:34.729 INF load_config:282 |     "top_k": 64,
+15:04:34.729 INF load_config:282 |     "top_p": 0.95
+15:04:34.729 INF load_config:282 | }
+15:04:34.729 INF Init:1139 | LLM init ok
+Commands:
+  /q, /exit  退出
+  /reset     重置 kvcache
+  /dd        删除一轮对话
+  /pp        打印历史对话
+Ctrl+C: 停止当前生成
+VLM enabled: after each prompt, input media path (empty = text-only). Use "video:<frames_dir>" for video, "audio:<file>" for reserved audio placeholder.
+----------------------------------------
+prompt >> who are you?
+media >>
+15:04:39.368 INF SetKVCache:1437 | decode_grpid:0 prefill_grpid:1 history_cap:0 total_cap:128 symbolic_cap:1 precompute_len:0 input_num_token:24 prefer_symbolic_group:0
+15:04:39.368 INF SetKVCache:1458 | current prefill_max_token_num:1152
+15:04:39.408 INF SetKVCache:1462 | first run
+15:04:39.409 INF Run:1553 | input token num : 24, prefill_split_num : 1
+15:04:39.482 INF Run:1640 | prefill chunk p=0 history_len=0 grpid=1 kv_cache_num=0 input_tokens=24
+15:04:39.483 INF Run:1665 | prefill indices shape: p=0 idx_elems=128 idx_rows=1 pos_rows=0
+15:04:39.764 INF Run:1837 | ttft: 355.37 ms
+I am Gemma 4, a Large Language Model developed by Google DeepMind. I am an open weights model.
+
+15:04:44.087 NTC Run:2103 | hit eos,decode avg 5.09 token/s
+15:04:44.088 INF GetKVCache:1408 | precompute_len:47, remaining:1105
+prompt >> Please describe the image in detail.
+media >> /root/yongqiang/auto_model_deployment/gemma-4-E2B-it/assets/sample.png
+15:06:14.416 INF EncodeForContent:1122 | vision cache hit (disk): /root/yongqiang/auto_model_deployment/gemma-4-E2B-it/assets/sample.png
+15:06:14.416 INF EncodeForContent:1131 | vision cache hit (mem): /root/yongqiang/auto_model_deployment/gemma-4-E2B-it/assets/sample.png
+15:06:14.419 INF SetKVCache:1437 | decode_grpid:0 prefill_grpid:3 history_cap:256 total_cap:384 symbolic_cap:256 precompute_len:47 input_num_token:94 prefer_symbolic_group:1
+15:06:14.419 INF SetKVCache:1458 | current prefill_max_token_num:1024
+15:06:14.429 INF Run:1553 | input token num : 94, prefill_split_num : 1
+15:06:14.703 INF Run:1640 | prefill chunk p=0 history_len=47 grpid=3 kv_cache_num=256 input_tokens=94
+15:06:14.703 INF Run:1665 | prefill indices shape: p=0 idx_elems=128 idx_rows=1 pos_rows=0
+15:06:15.027 INF Run:1837 | ttft: 597.95 ms
+I see an image of a cartoon character that resembles a cooked or stylized lobster.
+
+Here is a detailed description of the image:
+
+*   **Subject:** The central subject is a bright red, stylized lobster.
+*   **Style:** The illustration is highly cartoonish and vibrant, featuring thick outlines and bright, saturated colors, suggesting a playful or energetic style.
+*   **Features:**
+    *   The lobster has large, expressive eyes and a wide, toothy grin, giving it a mischievous or energetic expression.
+    *   Its claws (pincers) are prominent and stylized.
+    *   The body is segmented, typical of a lobster, but rendered in a simplified, bold manner.
+    *   It has a curved, slightly exaggerated posture.
+*   **Outline/Background:** The character is set against a plain white background. The image has a glossy or sticker-like finish, indicated by a slight shadow effect or outline around the character, suggesting it might be a graphic or sticker design.
+*   **Overall Impression:** The image is energetic, bold, and fun, clearly designed as a mascot or a character illustration.
+
+15:07:09.593 NTC Run:2103 | hit eos,decode avg 4.33 token/s
+15:07:09.593 INF GetKVCache:1408 | precompute_len:378, remaining:774
+```
+
+
+### Serve with `axllm`
+
+To launch the packaged model through the local `axllm` service:
+
+Note: the command below assumes you run it from the parent directory of `AXERA-TECH/gemma-4-E2B-it`. If you are already inside the package directory, use `axllm serve . --port 8000` instead.
+
+```bash
+$ axllm serve AXERA-TECH/gemma-4-E2B-it --port 8000
+# output log example:
+16:22:21.336 INF Init:890 | LLM init start
+16:22:21.336 INF Init:905 | shared kv enabled: num_kv_shared_layers=20
+tokenizer_type = 3
+huggingface tokenizer mode = space_replace_bpe
+ 13% | ####                             |   5 /  38 [10.08s<76.64s, 0.50 count/s] init 3 axmodel ok,remain_cmm(4704 MB                                                                                                                       15% | #####                            |   6 /  38 [12.32s<78.01s, 0.49 count/s] init 4 axmodel ok,remain_cmm(4635 MB                                                                                                                       18% | #####                            |   7 /  38 [17.98s<97.59s, 0.39 count/s] init 5 axmodel ok,remain_cmm(4580 MB                                                                                                                       21% | ######                           |   8 /  38 [18.54s<88.05s, 0.43 count/s] init 6 axmodel ok,remain_cmm(4525 MB                                                                                                                       23% | #######                          |   9 /  38 [19.06s<80.49s, 0.47 count/s] init 7 axmodel ok,remain_cmm(4470 MB                                                                                                                       26% | ########                         |  10 /  38 [19.66s<74.70s, 0.51 count/s] init 8 axmodel ok,remain_cmm(4415 MB                                                                                                                       28% | #########                        |  11 /  38 [20.41s<70.49s, 0.54 count/s] init 9 axmodel ok,remain_cmm(4346 MB                                                                                                                       31% | ##########                       |  12 /  38 [20.81s<65.91s, 0.58 count/s] init 10 axmodel ok,remain_cmm(4291 M                                                                                                                       34% | ##########                       |  13 /  38 [21.22s<62.04s, 0.61 count/s] init 11 axmodel ok,remain_cmm(4236 M                                                                                                                       36% | ###########                      |  14 /  38 [21.89s<59.42s, 0.64 count/s] init 12 axmodel ok,remain_cmm(4182 M                                                                                                                       39% | ############                     |  15 /  38 [22.24s<56.34s, 0.67 count/s] init 13 axmodel ok,remain_cmm(4127 M                                                                                                                       42% | #############                    |  16 /  38 [22.56s<53.57s, 0.71 count/s] init 14 axmodel ok,remain_cmm(4057 M                                                                                                                       44% | ##############                   |  17 /  38 [23.11s<51.66s, 0.74 count/s] init 15 axmodel ok,remain_cmm(3972 M                                                                                                                       47% | ###############                  |  18 /  38 [23.54s<49.69s, 0.76 count/s] init 16 axmodel ok,remain_cmm(3887 M                                                                                                                       50% | ################                 |  19 /  38 [24.28s<48.56s, 0.78 count/s] init 17 axmodel ok,remain_cmm(3801 M                                                                                                                       52% | ################                 |  20 /  38 [24.63s<46.80s, 0.81 count/s] init 18 axmodel ok,remain_cmm(3716 M                                                                                                                       55% | #################                |  21 /  38 [24.92s<45.08s, 0.84 count/s] init 19 axmodel ok,remain_cmm(3617 M                                                                                                                       57% | ##################               |  22 /  38 [25.67s<44.33s, 0.86 count/s] init 20 axmodel ok,remain_cmm(3532 M                                                                                                                       60% | ###################              |  23 /  38 [26.33s<43.50s, 0.87 count/s] init 21 axmodel ok,remain_cmm(3447 M                                                                                                                       63% | ####################             |  24 /  38 [27.17s<43.02s, 0.88 count/s] init 22 axmodel ok,remain_cmm(3361 M                                                                                                                       65% | #####################            |  25 /  38 [28.33s<43.06s, 0.88 count/s] init 23 axmodel ok,remain_cmm(3276 M                                                                                                                       68% | #####################            |  26 /  38 [29.70s<43.41s, 0.88 count/s] init 24 axmodel ok,remain_cmm(3177 M                                                                                                                       71% | ######################           |  27 /  38 [30.89s<43.48s, 0.87 count/s] init 25 axmodel ok,remain_cmm(3092 M                                                                                                                       73% | #######################          |  28 /  38 [32.16s<43.65s, 0.87 count/s] init 26 axmodel ok,remain_cmm(3006 M                                                                                                                       76% | ########################         |  29 /  38 [33.32s<43.67s, 0.87 count/s] init 27 axmodel ok,remain_cmm(2921 M                                                                                                                       78% | #########################        |  30 /  38 [34.43s<43.61s, 0.87 count/s] init 28 axmodel ok,remain_cmm(2836 M                                                                                                                       81% | ##########################       |  31 /  38 [35.69s<43.75s, 0.87 count/s] init 29 axmodel ok,remain_cmm(2737 M                                                                                                                       84% | ##########################       |  32 /  38 [36.84s<43.75s, 0.87 count/s] init 30 axmodel ok,remain_cmm(2652 M                                                                                                                       86% | ###########################      |  33 /  38 [37.75s<43.47s, 0.87 count/s] init 31 axmodel ok,remain_cmm(2566 M                                                                                                                       89% | ############################     |  34 /  38 [38.44s<42.96s, 0.88 count/s] init 32 axmodel ok,remain_cmm(2481 M                                                                                                                       92% | #############################    |  35 /  38 [39.06s<42.41s, 0.90 count/s] init 33 axmodel ok,remain_cmm(2396 M                                                                                                                       94% | ##############################   |  36 /  38 [39.44s<41.63s, 0.91 count/s] init 34 axmodel ok,remain_cmm(2297 M                                                                                                                       97% | ###############################  |  37 /  38 [41.12s<42.23s, 0.90 count/s] init post axmodel ok,remain_cmm(1877 MB)
+16:23:02.455 INF Init:1045 | max_token_len : 2047
+16:23:02.455 INF Init:1048 | kv_cache_size : 256, kv_cache_num: 2047
+16:23:02.455 INF init_groups_from_model:606 | prefill_token_num : 128
+16:23:02.455 INF init_groups_from_model:820 | decode grp: 0, gid: 0, max_token_len : 2047
+16:23:02.455 INF init_groups_from_model:824 | prefill grp: 0, gid: 1, history_cap: 0, total_cap: 128, symbolic_cap: 1
+16:23:02.455 INF init_groups_from_model:824 | prefill grp: 1, gid: 2, history_cap: 128, total_cap: 256, symbolic_cap: 128
+16:23:02.455 INF init_groups_from_model:824 | prefill grp: 2, gid: 3, history_cap: 256, total_cap: 384, symbolic_cap: 256
+16:23:02.455 INF init_groups_from_model:824 | prefill grp: 3, gid: 4, history_cap: 384, total_cap: 512, symbolic_cap: 384
+16:23:02.455 INF init_groups_from_model:824 | prefill grp: 4, gid: 5, history_cap: 512, total_cap: 640, symbolic_cap: 512
+16:23:02.455 INF init_groups_from_model:824 | prefill grp: 5, gid: 6, history_cap: 640, total_cap: 768, symbolic_cap: 640
+16:23:02.455 INF init_groups_from_model:824 | prefill grp: 6, gid: 7, history_cap: 768, total_cap: 896, symbolic_cap: 768
+16:23:02.455 INF init_groups_from_model:824 | prefill grp: 7, gid: 8, history_cap: 896, total_cap: 1024, symbolic_cap: 896
+16:23:02.455 INF init_groups_from_model:824 | prefill grp: 8, gid: 9, history_cap: 1024, total_cap: 1152, symbolic_cap: 1024
+16:23:02.455 INF init_groups_from_model:831 | prefill_max_token_num : 1152
+16:23:02.455 INF Init:27 | LLaMaEmbedSelector use mmap
+100% | ################################ |  38 /  38 [41.12s<41.12s, 0.92 count/s] embed_selector init ok
+16:23:02.472 INF Init:475 | Gemma4 per-layer helper enabled: vocab=262144 hidden=1536 layers=35 per_layer=256 pad=0
+16:23:03.400 INF Init:785 | Gemma4-VL token ids: image_pad=258880 video_pad=258884
+16:23:03.400 INF Init:792 | VisionModule init ok: type=Gemma4VL, tokens_per_block=70, embed_size=1536, out_dtype=fp32
+16:23:03.400 WRN Init:801 | Vision preprocess backend: SimpleCV (OpenCV not found at build time; minor differences vs OpenCV are possible)
+16:23:03.404 INF load_config:282 | load config:
+16:23:03.404 INF load_config:282 | {
+16:23:03.404 INF load_config:282 |     "enable_repetition_penalty": false,
+16:23:03.404 INF load_config:282 |     "enable_temperature": true,
+16:23:03.404 INF load_config:282 |     "enable_top_k_sampling": false,
+16:23:03.404 INF load_config:282 |     "enable_top_p_sampling": true,
+16:23:03.404 INF load_config:282 |     "penalty_window": 64,
+16:23:03.404 INF load_config:282 |     "repetition_penalty": 1.0,
+16:23:03.404 INF load_config:282 |     "temperature": 1.0,
+16:23:03.404 INF load_config:282 |     "top_k": 64,
+16:23:03.404 INF load_config:282 |     "top_p": 0.95
+16:23:03.404 INF load_config:282 | }
+16:23:03.404 INF Init:1139 | LLM init ok
+Starting server on port 8000 with model 'AXERA-TECH/gemma-4-E2B-it'...
+API URLs:
+  GET  http://127.0.0.1:8000/health
+  GET  http://127.0.0.1:8000/v1/models
+  POST http://127.0.0.1:8000/v1/chat/completions
+  GET  http://10.168.232.217:8000/health
+  GET  http://10.168.232.217:8000/v1/models
+  POST http://10.168.232.217:8000/v1/chat/completions
+  GET  http://172.17.0.1:8000/health
+  GET  http://172.17.0.1:8000/v1/models
+  POST http://172.17.0.1:8000/v1/chat/completions
+Aliases:
+  GET  http://127.0.0.1:8000/models
+  POST http://127.0.0.1:8000/chat/completions
+  GET  http://10.168.232.217:8000/models
+  POST http://10.168.232.217:8000/chat/completions
+  GET  http://172.17.0.1:8000/models
+  POST http://172.17.0.1:8000/chat/completions
+OpenAI API Server starting on http://0.0.0.0:8000
+Max concurrency: 1
+Models: AXERA-TECH/gemma-4-E2B-it
+```
+
+You can then send requests to the server using the API endpoints shown in the log. For example, to check the health status and list the available models:
+
+```sh
+$ curl http://127.0.0.1:8000/health
+$ curl http://127.0.0.1:8000/v1/models
+
+# Example output:
+root@ax650 ~ # curl http://127.0.0.1:8000/health
+{
+  "concurrency": 0,
+  "max_concurrency": 1,
+  "status": "healthy"
+}
+root@ax650 ~ # curl http://127.0.0.1:8000/v1/models
+{
+  "data": [
+    {
+      "created": 1777019000,
+      "id": "AXERA-TECH/gemma-4-E2B-it",
+      "object": "model",
+      "owned_by": "openai-api"
+    }
+  ],
+  "object": "list"
+}
+```
+
+## Python Runtime Requirements
 
 Install the following packages on the AX board:
 
@@ -88,15 +338,18 @@ Install the following packages on the AX board:
 - `numpy`
 - `ml_dtypes`
 - `pillow`
+- `torch`
 - `gradio` for the web demo only
 
-If your board image ships with an older `transformers` stack, you can use a pure-Python overlay directory instead:
+Before running any Python demo command in this package, make sure the Python dependency overlay is visible in `PYTHONPATH`:
 
 ```bash
 export PYTHONPATH=/path/to/your/gemma4_pydeps:$PYTHONPATH
 ```
 
-## Quick Start
+If your board image ships with an older `transformers` stack, this pure-Python overlay is the recommended way to supply the required runtime dependencies.
+
+## Legacy Python Demo Flow
 
 Enter the package directory on the board:
 
@@ -130,9 +383,7 @@ answer >> The capital of the United States is **Washington, D.C.**
 
 ### Multimodal Inference
 
-Use the sample image shown below: `assets/sample.png`
-
-![sample](assets/sample.png)
+Use the sample image shown above: `assets/sample.png`
 
 Recommended profile: `70` soft tokens at `336x480`.
 
@@ -173,13 +424,13 @@ Here's a detailed description:
 **Overall Impression:** The image is energetic, bold, and eye-catching, suitable for use as a mascot, icon, or graphic design element.
 ```
 
-The package also includes two additional higher-resolution Vision models:
+In addition to the default `t70` profile, the package also includes two higher-resolution Vision models:
 
 | VIT file | Resolution | Soft tokens |
 | --- | --- | --- |
-| `vit_models/gemma4_vision_h336_w480_t70.axmodel` | `336x480` | `70` |
-| `vit_models/gemma4_vision_h480_w672_t140.axmodel` | `480x672` | `140` |
-| `vit_models/gemma4_vision_h672_w960_t280.axmodel` | `672x960` | `280` |
+| `gemma4_vision_h336_w480_t70.axmodel` | `336x480` | `70` |
+| `gemma4_vision_h480_w672_t140.axmodel` | `480x672` | `140` |
+| `gemma4_vision_h672_w960_t280.axmodel` | `672x960` | `280` |
 
 To use a different profile, pass `--vit_model_path` explicitly. The runtime will infer the matching soft-token count from the filename:
 
@@ -188,7 +439,7 @@ python3 infer_axmodel.py \
   --image_path ./assets/sample.png \
   --prompt "Describe this image in detail." \
   --system_prompt "" \
-  --vit_model_path ./vit_models/gemma4_vision_h480_w672_t140.axmodel \
+  --vit_model_path ./gemma4_vision_h480_w672_t140.axmodel \
   --max_new_tokens 256
 ```
 
@@ -197,7 +448,7 @@ python3 infer_axmodel.py \
   --image_path ./assets/sample.png \
   --prompt "Describe this image in detail." \
   --system_prompt "" \
-  --vit_model_path ./vit_models/gemma4_vision_h672_w960_t280.axmodel \
+  --vit_model_path ./gemma4_vision_h672_w960_t280.axmodel \
   --max_new_tokens 1024
 ```
 
@@ -243,23 +494,26 @@ python3 gradio_demo.py \
 
 After the server starts, open `http://<board-ip>:7860` in your browser.
 
-## Packaged Runtime Paths
+## Packaged Python Runtime Paths
 
-The release package uses the following default paths:
+The Python demo scripts use the following default paths:
 
 - Tokenizer and config: `./gemma_4_e2b_it_tokenizer`
-- Text LLM axmodels: `./gemma_4_e2b_it_ax650n_axmodel`
-- Vision axmodels: `./vit_models`
+- Text LLM runtime root: `./`
+- Vision axmodels: `./`
 
 If you move any of these directories, pass the new values with `--hf_model`, `--axmodel_path`, and `--vit_model_path`.
 
-## Notes
+For the Python demo flow, `--axmodel_path` should point to the directory that contains the text runtime files such as `gemma4_text_p128_l*.axmodel`, `gemma4_text_post.axmodel`, `model.embed_tokens.weight.bfloat16.bin`, and the `model.*per_layer*.npy` files.
 
-- `.axmodel` execution is board-only and is not supported on x86 hosts.
-- The default multimodal profile uses `70` image soft tokens and matches the packaged `336x480` Vision model.
-- The current text runtime package contains `35` decoder layers and `kv_cache_len=2047`.
-- The packaged runtime already includes the embedding and per-layer weight files needed by Gemma 4. Original `model.safetensors` weights are not required for board-side inference.
-- Files under `assets/` are demo inputs for inference examples.
+These path arguments apply to the Python demo flow only. The `axllm` flow reads the same root-level runtime files packaged in this repository.
+
+## Conversion References
+
+If you need the original model files or want to rebuild the deployment artifacts, start with:
+
+- Original Hugging Face model: [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it)
+- AXERA conversion and deployment workflow: [AXERA-TECH/gemma-4-E2B-it.axera](https://github.com/AXERA-TECH/gemma-4-E2B-it.axera)
 
 ## Discussion
 

config.json

@@ -1 +1,75 @@
-{}
+{
+  "system_prompt": "You are a helpful assistant.",
+  "model_name": "AXERA-TECH/gemma-4-E2B-it",
+  "url_tokenizer_model": "gemma4_tokenizer.txt",
+  "tokenizer_type": "Gemma4VL",
+  "post_config_path": "post_config.json",
+  "template_filename_axmodel": "gemma4_text_p128_l%d_together.axmodel",
+  "axmodel_num": 35,
+  "filename_post_axmodel": "gemma4_text_post.axmodel",
+  "filename_tokens_embed": "model.embed_tokens.weight.bfloat16.bin",
+  "tokens_embed_num": 262144,
+  "tokens_embed_size": 1536,
+  "text_config": {
+    "hidden_size": 1536,
+    "num_hidden_layers": 35,
+    "num_key_value_heads": 1,
+    "head_dim": 256,
+    "global_head_dim": 512,
+    "num_kv_shared_layers": 20,
+    "layer_types": [
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "sliding_attention",
+      "full_attention"
+    ]
+  },
+  "pad_token_id": 0,
+  "hidden_size_per_layer_input": 256,
+  "rms_norm_eps": 1e-06,
+  "filename_tokens_embed_per_layer": "model.embed_tokens_per_layer.weight.npy",
+  "filename_per_layer_model_projection": "model.per_layer_model_projection.weight.npy",
+  "filename_per_layer_projection_norm": "model.per_layer_projection_norm.weight.npy",
+  "use_mmap_load_embed": true,
+  "vlm_type": "Gemma4VL",
+  "filename_image_encoder_axmodel": "gemma4_vision_h336_w480_t70.axmodel",
+  "vision_width": 480,
+  "vision_height": 336,
+  "vision_patch_size": 16,
+  "vision_cache_dir": "vision_cache",
+  "use_mmap_load_layer": true,
+  "devices": [
+    0
+  ]
+}

gemma_4_e2b_it_ax650n_axmodel/model.embed_tokens.weight.npy → gemma4_tokenizer.txt

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:39202a6627a585f06b0852d267311d3a063fb8d610c1c106ec403d05d42afba0
-size 1610612864
+oid sha256:90603c2c15f0d202d63c5c7767e4787e7e3909d74ee2c47f93914d3575dbd0ef
+size 17165772

gradio_demo.py

@@ -23,6 +23,8 @@ from utils.gemma4_multimodal import resolve_resize
 from utils.gemma4_multimodal import resize_image
 from utils.gemma4_multimodal import to_numpy_fp32
 from utils.infer_func import InferManager
+from utils.runtime_layout import default_axmodel_path
+from utils.runtime_layout import load_text_embeddings
 from utils.vision_output import describe_output_shapes
 from utils.vision_output import select_vit_output
 
@@ -53,14 +55,7 @@ def _default_hf_model() -> str:
 
 def _default_axmodel_path() -> str:
     script_dir = Path(__file__).resolve().parent
-    for candidate in [
-        script_dir / "gemma_4_e2b_it_ax650n_axmodel",
-        script_dir / "gemma_4_e2b_it_ax650n_w4a16_axmodel",
-        script_dir / "gemma-4-E2B-it_axmodel",
-    ]:
-        if candidate.exists():
-            return str(candidate)
-    return str(script_dir / "gemma_4_e2b_it_ax650n_axmodel")
+    return default_axmodel_path(script_dir)
 
 
 def _default_vit_model_path() -> str:
@@ -68,7 +63,9 @@ def _default_vit_model_path() -> str:
     resize_h, resize_w, expected_tokens = resolve_resize(DEFAULT_MAX_SOFT_TOKENS)
     stem = f"gemma4_vision_h{resize_h}_w{resize_w}_t{expected_tokens}"
     candidates = [
+        script_dir / f"{stem}.axmodel",
         script_dir / "vit_models" / f"{stem}.axmodel",
+        script_dir / f"{stem}.onnx",
         script_dir / "vit_models" / f"{stem}.onnx",
         script_dir.parent / "model_convert" / "compiled_output" / f"{stem}.axmodel",
         script_dir.parent / "model_convert" / "vit-models" / f"{stem}.onnx",
@@ -108,7 +105,7 @@ class Gemma4GradioDemo:
         self.processor = load_processor(hf_model)
         self.tokenizer = self.processor.tokenizer
         self.config = load_text_runtime_config(hf_model)
-        self.embeds = np.load(os.path.join(axmodel_path, "model.embed_tokens.weight.npy"))
+        self.embeds = load_text_embeddings(axmodel_path, self.config)
         self.axmodel_path = axmodel_path
         self.vit_model_path = vit_model_path
         self.max_seq_len = max_seq_len
@@ -345,7 +342,7 @@ def main():
     parser.add_argument("--hf_model", type=str, default=_default_hf_model(),
                         help="Path to Gemma 4 tokenizer/config directory")
     parser.add_argument("--axmodel_path", type=str, default=_default_axmodel_path(),
-                        help="Path to compiled LLM axmodel folder")
+                        help="Path to the packaged LLM runtime root or legacy axmodel folder")
     parser.add_argument("--vit_model_path", type=str, default=_default_vit_model_path(),
                         help="Path to Gemma 4 vision ONNX model or .axmodel")
     parser.add_argument("--port", type=int, default=7860, help="Gradio server port")

infer_axmodel.py

@@ -17,6 +17,8 @@ from utils.gemma4_multimodal import replace_image_tokens
 from utils.gemma4_multimodal import resolve_resize
 from utils.gemma4_multimodal import to_numpy_fp32
 from utils.gemma4_per_layer import Gemma4PerLayerInputs
+from utils.runtime_layout import default_axmodel_path
+from utils.runtime_layout import load_text_embeddings
 from utils.infer_func import InferManager
 from utils.vision_output import describe_output_shapes
 from utils.vision_output import select_vit_output
@@ -36,15 +38,7 @@ def _default_hf_model() -> str:
 
 def _default_axmodel_path() -> str:
     script_dir = Path(__file__).resolve().parent
-    candidates = [
-        script_dir / "gemma_4_e2b_it_ax650n_axmodel",
-        script_dir / "gemma_4_e2b_it_ax650n_w4a16_axmodel",
-        script_dir / "gemma-4-E2B-it_axmodel",
-    ]
-    for candidate in candidates:
-        if candidate.exists():
-            return str(candidate)
-    return str(candidates[0])
+    return default_axmodel_path(script_dir)
 
 
 def _default_vit_model_path() -> str:
@@ -52,7 +46,9 @@ def _default_vit_model_path() -> str:
     resize_h, resize_w, expected_tokens = resolve_resize(DEFAULT_MAX_SOFT_TOKENS)
     stem = f"gemma4_vision_h{resize_h}_w{resize_w}_t{expected_tokens}"
     candidates = [
+        script_dir / f"{stem}.axmodel",
         script_dir / "vit_models" / f"{stem}.axmodel",
+        script_dir / f"{stem}.onnx",
         script_dir / "vit_models" / f"{stem}.onnx",
         script_dir.parent / "model_convert" / "compiled_output" / f"{stem}.axmodel",
         script_dir.parent / "model_convert" / "vit-models" / f"{stem}.onnx",
@@ -111,7 +107,7 @@ if __name__ == "__main__":
     parser.add_argument("--hf_model", type=str, default=_default_hf_model(),
                         help="Path to Gemma 4 tokenizer/config directory")
     parser.add_argument("--axmodel_path", type=str, default=_default_axmodel_path(),
-                        help="Path to compiled LLM axmodel folder")
+                        help="Path to the packaged LLM runtime root or legacy axmodel folder")
     parser.add_argument("--vit_model_path", type=str, default=_default_vit_model_path(),
                         help="Path to Gemma 4 vision ONNX model or .axmodel")
     parser.add_argument("--image_path", type=str, default="",
@@ -135,7 +131,7 @@ if __name__ == "__main__":
     args = parser.parse_args()
 
     config = load_text_runtime_config(args.hf_model)
-    embeds = np.load(os.path.join(args.axmodel_path, "model.embed_tokens.weight.npy"))
+    embeds = load_text_embeddings(args.axmodel_path, config)
     per_layer_helper = None
     if int(getattr(config, "hidden_size_per_layer_input", 0) or 0) > 0:
         per_layer_helper = Gemma4PerLayerInputs(args.axmodel_path, config)
@@ -147,6 +143,9 @@ if __name__ == "__main__":
             print(f"[INFO] Auto-detected max_soft_tokens={detected} from VIT model: {args.vit_model_path}")
             args.max_soft_tokens = detected
 
+    kv_cache_len = int(getattr(config, "kv_cache_len", 2047) or 2047)
+    imer = InferManager(config, args.axmodel_path, max_seq_len=kv_cache_len, per_layer_helper=per_layer_helper)
+
     mm_token_type_ids = None
     prefill_per_layer_inputs = None
     if args.image_path:
@@ -233,8 +232,6 @@ if __name__ == "__main__":
 
     eos_token_id = config.eos_token_id if isinstance(config.eos_token_id, list) else None
 
-    kv_cache_len = int(getattr(config, "kv_cache_len", 2047) or 2047)
-    imer = InferManager(config, args.axmodel_path, max_seq_len=kv_cache_len, per_layer_helper=per_layer_helper)
     token_ids = imer.prefill(
         tokenizer,
         token_ids,