---
library_name: transformers
license: gemma
base_model:
  - google/gemma-4-E2B-it
tags:
  - gemma4
  - multimodal
  - axera
  - ax650
language:
  - en
---

# Gemma 4 E2B on AXERA NPU

Ready-to-run AX650/NPU3 deployment package for `google/gemma-4-E2B-it`.

- Includes the tokenizer/config files required at runtime.
- Includes compiled Gemma 4 text `.axmodel` files and Vision `.axmodel` files.
- Supports both text-only chat and single-image multimodal inference.
- For model export or recompilation, see [AXERA-TECH/gemma-4-E2B-it.axera](https://github.com/AXERA-TECH/gemma-4-E2B-it.axera).

## Supported Platform

- [x] AX650 / NPU3

## Performance

Measured on AX650 / NPU3. `TTFT` stands for time to first token.

- `w8a16`: TTFT is approximately `1664 ms`, with a decode throughput of approximately `10.44 tokens/s`.
- `w4a16`: TTFT is approximately `1233.7 ms`, with a decode throughput of approximately `15.22 tokens/s`.

The packaged text runtime in this release is the `w4a16` variant under `gemma_4_e2b_it_ax650n_w4a16_axmodel`. The `w8a16` numbers are included for reference.

## Vision Encoder Latency

| Model | Resolution | Soft Tokens | Time (ms) |
|---|---|---|---|
| `gemma4_vision_h336_w480_t70.axmodel` | 336×480 | 70 | 87.966 ms |
| `gemma4_vision_h480_w672_t140.axmodel` | 480×672 | 140 | 258.329 ms |
| `gemma4_vision_h672_w960_t280.axmodel` | 672×960 | 280 | 750.429 ms |

## Package Layout

```text
.
├── README.md
├── config.json
├── infer_axmodel.py
├── gradio_demo.py
├── assets/
├── gemma_4_e2b_it_tokenizer/
├── gemma_4_e2b_it_ax650n_w4a16_axmodel/
├── vit_models/
└── utils/
```

The runtime scripts auto-detect the packaged directories above. If you keep the release layout unchanged, you can run the examples below without passing extra path arguments.

## Runtime Requirements

Install the following packages on the AX board:

- `pyaxengine`
- `transformers>=5.5.0`
- `numpy`
- `ml_dtypes`
- `pillow`
- `gradio` for the web demo only

If the board image ships with an older `transformers` stack, you can use a pure-Python overlay directory instead:

```bash
export PYTHONPATH=~/yongqiang/push_hugging_face/gemma4_pydeps:$PYTHONPATH
```

## Quick Start

Enter the release package directory on the board:

```bash
cd ~/yongqiang/push_hugging_face/gemma-4-E2B-it-GPTQ-INT4
```

### Text-Only Inference

```bash
python3 infer_axmodel.py \
  --prompt "What is the capital of the United States?" \
  --max_new_tokens 256
```

### Multimodal Inference

Recommended profile: `70` soft tokens at `336x480`.

```bash
python3 infer_axmodel.py \
  --image_path ./assets/sample_1.png \
  --prompt "Describe this image in detail." \
  --system_prompt "" \
  --max_new_tokens 256
```

The package also includes two higher-resolution Vision models:

| VIT file | Resolution | Soft tokens |
| --- | --- | --- |
| `vit_models/gemma4_vision_h336_w480_t70.axmodel` | `336x480` | `70` |
| `vit_models/gemma4_vision_h480_w672_t140.axmodel` | `480x672` | `140` |
| `vit_models/gemma4_vision_h672_w960_t280.axmodel` | `672x960` | `280` |

To switch to another profile, pass `--vit_model_path` explicitly. The runtime will infer the matching soft-token count from the filename:

```bash
python3 infer_axmodel.py \
  --image_path ./assets/sample_1.png \
  --prompt "Describe this image in detail." \
  --system_prompt "" \
  --vit_model_path ./vit_models/gemma4_vision_h480_w672_t140.axmodel \
  --max_new_tokens 256
```

```bash
python3 infer_axmodel.py \
  --image_path ./assets/sample_1.png \
  --prompt "Describe this image in detail." \
  --system_prompt "" \
  --vit_model_path ./vit_models/gemma4_vision_h672_w960_t280.axmodel \
  --max_new_tokens 256
```

### Gradio Demo

```bash
python3 gradio_demo.py \
  --host 0.0.0.0 \
  --port 7860
```

Then open `http://<board-ip>:7860` from your browser.

## Packaged Runtime Paths

The release package uses these default paths:

- Tokenizer and config: `./gemma_4_e2b_it_tokenizer`
- Text LLM axmodels: `./gemma_4_e2b_it_ax650n_w4a16_axmodel`
- Vision axmodels: `./vit_models`

If you move these directories, pass the new values with `--hf_model`, `--axmodel_path`, and `--vit_model_path`.

## Notes

- `.axmodel` execution is board-only and is not supported on x86 hosts.
- The default multimodal profile uses `70` image soft tokens, which matches the packaged `336x480` Vision model.
- The current text runtime package contains `35` decoder layers and `kv_cache_len=2047`.
- The packaged runtime already includes the embedding and per-layer weight files needed by Gemma 4. Original `model.safetensors` weights are not required for board-side inference.
- Files under `assets/` are demo inputs for inference examples.

## Discussion

- GitHub Issues
- QQ group: `139953715`