---
license: apache-2.0
base_model:
- mistralai/Devstral-Small-2507
pipeline_tag: text-generation
tags:
- code
---
# Devstral-Small-2507-AWQ

This model was forked in an attempt to make changes so it will run in vLLM.

## Method
Quantised using [casper-hansen/AutoAWQ](https://github.com/casper-hansen/AutoAWQ.git) and the following configs:
```
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
```

## Inference
The quantised model's configs and weights are stored in hf and safetensors format, but the tokeniser remains in mistral format. Please load inference arguments accordingly, e.g.,:

### vllm
```
vllm serve cpatonn/Devstral-Small-2507-AWQ --tokenizer_mode mistral --config_format hf --load_format safetensors --tool-call-parser mistral --enable-auto-tool-choice
```