| --- |
| language: |
| - en |
| - fr |
| - de |
| - it |
| - pt |
| - hi |
| - es |
| - th |
| base_model: |
| - deepseek-ai/DeepSeek-R1-Distill-Llama-8B |
| tags: |
| - directml |
| - windows |
| - onnx |
| - llama |
| - conversational |
| pipeline_tag: text-generation |
| --- |
| # Model Card for Model ID |
|
|
| ## Model Details |
| deepseek-ai/DeepSeek-R1-Distill-Llama-8B quantized to ONNX GenAI INT4 with Microsoft DirectML optimization.<br> |
| Output is reformatted that each sentence starts at new line to improve readability.<br> |
| <pre> |
| ... |
| vNewDecoded = tokenizer_stream.decode(new_token) |
| if re.findall("^[\x2E\x3A\x3B]$", vPreviousDecoded) and vNewDecoded.startswith(" ") and (not vNewDecoded.startswith(" *")) : |
| vNewDecoded = "\n" + vNewDecoded.replace(" ", "", 1) |
| print(vNewDecoded, end='', flush=True) |
| vPreviousDecoded = vNewDecoded |
| ... |
| </pre> |
| Output will start with COTS/reasoning.<br> |
| In the tokenizer_config.json, the "unk_token" value is changed from null to "" |
| |
| <img src="https://zci.sourceforge.io/epub/dsllama31.png"> |
| |
| ### Model Description |
| deepseek-ai/DeepSeek-R1-Distill-Llama-8B quantized to ONNX GenAI INT4 with Microsoft DirectML optimization<br> |
| https://onnxruntime.ai/docs/genai/howto/install.html#directml |
| |
| Created using ONNX Runtime GenAI's builder.py<br> |
| https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/src/python/py/models/builder.py |
| |
| Build options:<br> |
| INT4 accuracy level: FP32 (float32) |
| |
| - **Developed by:** Mochamad Aris Zamroni |
| |
| ### Model Sources [optional] |
| https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B |
| |
| ### Direct Use |
| This is Microsoft Windows DirectML optimized model.<br> |
| It might not be working in ONNX execution provider other than DmlExecutionProvider.<br> |
| The needed python scripts are included in this repository |
| |
| Prerequisites:<br> |
| 1. Install Python 3.11 from Windows Store:<br> |
| https://apps.microsoft.com/search/publisher?name=Python+Software+Foundation |
| |
| 3. Open command line cmd.exe |
| |
| 4. Create python virtual environment, activate the environment then install onnxruntime-genai-directml<br> |
| mkdir c:\temp<br> |
| cd c:\temp<br> |
| python -m venv dmlgenai<br> |
| dmlgenai\Scripts\activate.bat<br> |
| pip install onnxruntime-genai-directml |
| |
| 5. Use the onnxgenairun.py to get chat interface.<br> |
| It is modified version of "https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/phi3-qa.py".<br> |
| The modification makes the text output changes to new line after "., :, and ;" to make the output easier to be read. |
| |
| rem Change directory to where model and script files is stored<br> |
| cd this_onnx_model_directory<br> |
| python onnxgenairun.py --help<br> |
| python onnxgenairun.py -m . -v -g |
| |
| 5. (Optional but recommended) Device specific optimization.<br> |
| a. Open "dml-device-specific-optim.py" with text editor and change the file path accordingly.<br> |
| b. Run the python script: python dml-device-specific-optim.py<br> |
| c. Rename the original model.onnx to other file name and put and rename the optimized onnx file from step 5.b to model.onnx file.<br> |
| d. Rerun step 4. |
| |
| #### Speeds, Sizes, Times [optional] |
| 15 token/s in Radeon 780M with 8GB pre-allocated RAM.<br> |
| Increase to 16 token/s with device specific optimized model.onnx.<br> |
| As comparison, LM Studio using GGUF INT4 model and VulkanML GPU acceleration runs at 13 token/s. |
| |
| #### Hardware |
| AMD Ryzen Zen4 7840U with integrated Radeon 780M GPU<br> |
| RAM 32GB<br> |
| |
| #### Software |
| Microsoft DirectML on Windows 10 |
| |
| ## Model Card Authors [optional] |
| Mochamad Aris Zamroni |
| |
| ## Model Card Contact |
| https://www.linkedin.com/in/zamroni/ |
