| --- |
| license: apache-2.0 |
| language: |
| - en |
| - ja |
| base_model: |
| - Rakuten/RakutenAI-7B |
| tags: |
| - onnx |
| - int4 |
| - quantized |
| - on-device |
| - mistral |
| --- |
| # RakutenAI-7B-chat-onnx |
|
|
| ## Model Description |
|
|
| RakutenAI-7B-chat-onnx is an INT4-quantized, ONNX-exported chat model derived from [RakutenAI-7B](https://huggingface.co/Rakuten/RakutenAI-7B), optimized for efficient on-device inference across a range of hardware execution providers. |
|
|
| Starting from the RakutenAI-7B base model, the model was developed through the following stages: |
|
|
| 1. **Continual pre-training (CPT)** — further pre-training of the base model to strengthen general and domain knowledge beyond the original checkpoint. |
| 2. **Supervised fine-tuning (SFT)** — instruction- and chat-tuning on curated conversational data. |
| 3. **Preference optimization** — aligning model responses to human preference data. |
|
|
| The resulting chat model was then quantized to **INT4** precision and exported to **ONNX**. |
|
|
| RakutenAI-7B uses the Mistral model architecture and is based on the [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) pre-trained checkpoint. Its vocabulary has been extended from 32k to 48k tokens to achieve a better character-per-token rate for Japanese. |
|
|
| *The technical report for the initial RakutenAI-7B suite can be accessed at [arXiv](https://arxiv.org/abs/2403.15484).* |
|
|
| ## Model Evaluation |
|
|
| ### Japanese |
|
|
| | Model Name | Avg. | JCS | JNLI | MARC-ja | JSQuAD | Jaqket v2 | XLSum-ja | xWino | MGSM | |
| |----------------------------|:-----:|:-------:|:-------:|:-------:|:-------:|:---------:|:--------:|:------:|:-------:| |
| | | | accuracy | accuracy | accuracy | exact-match | exact-match | rouge-2 | accuracy | accuracy | |
| | | | 3-shots | 3-shots | 3-shots | 2-shots | 1-shot | 1-shot | 0-shot | 5-shots | |
| | Rakuten/RakutenAI-7B-chat-onnx | 70.26 | 92.14 | 53.37 | 95.49 | 77.10 | 86.94 | 13.55 | 77.89 | 65.60 | |
| | [Rakuten/RakutenAI-7B-chat](https://huggingface.co/Rakuten/RakutenAI-7B-chat) | 58.80 | 79.27 | 81.64 | 72.34 | 60.63 | 69.24 | 12.02 | 71.22 | 24.00 | |
|
|
| <div style="text-align: center;">Table1: RakutenAI-7B-chat-onnx model performance on Japanese LM-Harness metrics in comparison with RakutenAI-7B-chat.</div> |
|
|
| ### English |
|
|
| | Model Name | Avg. | ARC | HellaSwag | MMLU | TruthfulQA | |
| |---------------------------------|:----------------:|:------------------------:|:------------------------:|:-----------------------:|:-----------------------:| |
| | | | accuracy | accuracy | accuracy | accuracy | |
| | | | 25-shots | 10-shots | 5-shots | 6-shots | |
| | Rakuten/RakutenAI-7B-chat-onnx | 54.42 | 43.09 | 65.65 | 54.28 | 54.64 | |
| | [Rakuten/RakutenAI-7B-chat](https://huggingface.co/Rakuten/RakutenAI-7B-chat) | 59.20 | 50.00 | 75.94 | 54.51 | 56.34 | |
|
|
| <div style="text-align: center;">Table2: RakutenAI-7B-chat-onnx model performance on English LM-Harness metrics in comparison with RakutenAI-7B-chat. </div> |
|
|
| ## Quantization for On-Device Inference |
|
|
| To enable efficient on-device deployment, the chat model was quantized to **INT4** and exported as separate ONNX builds targeting the following execution providers: |
|
|
| | Execution Provider | Target Hardware | Foundry Local Model ID | |
| |---|---|---| |
| | CPU EP | Generic CPU | `rakutenai-7b-chat-generic-cpu:1` | |
| | GPU EP | Generic GPU | `rakutenai-7b-chat-generic-gpu:1` | |
| | OpenVINO EP (GPU) | Intel GPU | `rakutenai-7b-chat-openvino-gpu:1` | |
| | OpenVINO EP (NPU) | Intel NPU | `rakutenai-7b-chat-openvino-npu:1` | |
| | VitisAI EP | AMD NPU | `rakutenai-7b-chat-vitisai-npu:1` | |
|
|
| Each build is provided in its own branch. |
|
|
| ### Usage with Foundry Local |
|
|
| These ONNX Runtime GenAI builds are compatible with [Microsoft Foundry Local](https://learn.microsoft.com/azure/ai-foundry/foundry-local), Microsoft's on-device inferencing runtime. Foundry Local runs on ONNX Runtime and manages CPU, GPU, OpenVINO, and VitisAI execution providers automatically, so any of the five variants above can be loaded and served through Foundry Local's CLI, SDKs, or OpenAI-compatible local REST endpoint. |
|
|
| To use these builds, point Foundry Local at the repo folder directly: |
|
|
| ```bash |
| # Point Foundry Local at the folder containing the EP variant subfolders |
| foundry cache cd ./RakutenAI-7B-chat-onnx |
| |
| # Confirm Foundry Local recognizes the variants |
| foundry cache ls |
| |
| # Load and chat interactively with a specific variant |
| foundry model run rakutenai-7b-chat-generic-gpu:1 |
| ``` |
|
|
| Check [Foundry Local docs](https://learn.microsoft.com/azure/ai-foundry/foundry-local) for the version you're targeting. |
|
|
|
|
| ## Model Details |
|
|
| * **Developed by**: [Rakuten Group, Inc.](https://ai.rakuten.com/) |
| * **Language(s)**: Japanese, English |
| * **License**: This model is licensed under [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0). |
| * **Derived From**: [RakutenAI-7B](https://huggingface.co/Rakuten/RakutenAI-7B) (base model), via continual pre-training, supervised fine-tuning, and preference optimization. |
| * **Quantization**: INT4 (weight-only). |
|
|
| ### Limitations and Bias |
|
|
| The suite of RakutenAI-7B models is capable of generating human-like text on a wide range of topics. However, like all LLMs, they have limitations and can produce biased, inaccurate, or unsafe outputs. Please exercise caution and judgement while interacting with them. |
|
|
| ## Citation |
| For citing our work on the suite of RakutenAI-7B models, please use: |
|
|
|
|
| ```bibtex |
| @misc{rakutengroup2026rakutenai7bchatonnx, |
| title={RakutenAI-7B-chat-onnx}, |
| year={2026}, |
| publisher={Hugging Face}, |
| howpublished={\url{https://huggingface.co/Rakuten/RakutenAI-7B-chat-onnx}} |
| } |
| |
| @misc{rakutengroup2024rakutenai7b, |
| title={RakutenAI-7B: Extending Large Language Models for Japanese}, |
| year={2024}, |
| eprint={2403.15484}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL} |
| } |
| ``` |
|
|
