---
library_name: rkllm
license: other
license_name: deepseek
license_link: https://github.com/deepseek-ai/deepseek-LLM/raw/refs/heads/main/LICENSE-MODEL
language:
- en
base_model:
- deepseek-ai/deepseek-llm-7b-chat
tags:
- rkllm
- rknn-llm
- rk3588
- rockchip
- edge-ai
- llm
- deepseek
pipeline_tag: text-generation
---
# DeepSeek-LLM-7B-Chat — RKLLM build for RK3588 boards

**Author:** @jamescallander  
**Source model:** [deepseek-ai/deepseek-llm-7b-chat](https://huggingface.co/deepseek-ai/deepseek-llm-7b-chat)  

**Target:** Rockchip RK3588 NPU via **RKNN-LLM Runtime**  

> This repository hosts a **conversion** of `DeepSeek-LLM-7B-Chat` for use on Rockchip RK3588 equipped single board computers (Orange Pi 5 plus, Radxa Rock 5b+, Banana Pi M7, etc.). Conversion was performed using [rknn-llm toolkit](https://github.com/airockchip/rknn-llm).

#### Conversion details

- RKLLM-Toolkit version: v1.2.1
- NPU driver: v0.9.8
- Python 3.11
- Quantization: w8a8_g128
- Output: single-file `.rkllm` artifact
  
## Intended use

This build is intended for experimentation and deployment of `DeepSeek-LLM-7B-Chat` on Rockchip RK3588-based SBCs.  
  
## Limitations

- Requires 9GB free memory.
- Conversion is quantized (w8a8_g128), so slight quality differences from the FP16 baseline may occur.  
- Tested on Orange Pi 5 Plus, Orange Pi 5 Max, and Radxa Rock 5b+; other platforms may not be supported. 

## Quick start (RK3588)

### 1) Install runtime

The RKNN-LLM toolkit and instructions can be found on the specific development board's manufacturer website or from [airockchip's github page](https://github.com/airockchip).

Download and install the required packages as per the toolkit's instructions.

### 2) Simple Flask server deployment

The simplest way the deploy the `.rkllm` converted model is using an example script provided in the toolkit in this directory: `rknn-llm/examples/rkllm_server_demo`

```bash
python3 <TOOLKIT_PATH>/rknn-llm/examples/rkllm_server_demo/flask_server.py \
  --rkllm_model_path <MODEL_PATH>/rk3588-deepseek-llm-7b-chat.rkllm \
  --target_platform rk3588
```

### 3) Sending a request

A basic format for message request is:

```json
{
    "model":"deepseek-7b",
    "messages":[{
        "role":"user",
        "content":"<YOUR_PROMPT_HERE>"}],
    "stream":false
}
```

 
Example request using `curl`: 

```bash
curl -s -X POST <MODEL_SERVER_IP_ADDRESS>:8080/rkllm_chat \
    -H 'Content-Type: application/json' \
    -d '{"model":"deepseek-7b","messages":[{"role":"user","content":"In one sentence, who was Napoleon?"}],"stream":false}'
```



The response is formated in the following way:

```json
{
    "choices":[{
        "finish_reason":"stop",
        "index":0,
        "logprobs":null,
        "message":{
            "content":"<MODEL_REPLY_HERE>",
            "role":"assistant"}}],
        "created":null,
        "id":"rkllm_chat",
        "object":"rkllm_chat",
        "usage":{
            "completion_tokens":null,
            "prompt_tokens":null,
            "total_tokens":null}
}
```



Example response:

```json
{"choices":[{"finish_reason":"stop","index":0,"logprobs":null,"message":{"content":"Napoleon Bonaparte (1769-1821) was a French military leader and statesman who rose to power during the French Revolution, becoming Emperor of France from 1804 to 1815 and implementing various reforms and conquests that had a lasting impact on European history.","role":"assistant"}}],"created":null,"id":"rkllm_chat","object":"rkllm_chat","usage":{"completion_tokens":null,"prompt_tokens":null,"total_tokens":null}}
```

### 4) UI compatibility

This server exposes an **OpenAI-compatible Chat Completions API**.

You can connect it to any OpenAI-compatible client or UI (for example: [Open WebUI](https://github.com/open-webui/open-webui?utm_source=chatgpt.com))
- Configure your client with the API base: `http://<SERVER_IP_ADDRESS>:8080` and use the endpoint: `/rkllm_chat`
- Make sure the `model` field matches the converted model’s name, for example:

```json
{
 "model": "DeepSeek-LLM-7B-Chat",
 "messages": [{"role":"user","content":"Hello!"}],
 "stream": false
}
```

# License

This conversion follows the license of the source model: [DeepSeek LLM license](https://github.com/deepseek-ai/deepseek-LLM/blob/main/LICENSE-MODEL).

- **Attribution:** Built with DeepSeek (© 2023 DeepSeek).  
- **Modifications:** quantization (w8a8_g128), export to `.rkllm` format for RK3588 SBCs.  
- **Use Restrictions:** The model and its derivatives may not be used for military purposes, harming minors, harassment, generating PII without authorization, fully automated binding decisions, or other prohibited uses listed in Attachment A of the DeepSeek License Agreement.


For more information on the deployment and use of `.rkllm` models on RK3588 platforms, please refer to the RKNN-LLM toolkit documentation.