How to use from
vLLMUse Docker
docker model run hf.co/btbtyler09/Devstral-Small-2507-AWQQuick Links
Devstral-Small-2507-AWQ
This model was forked in an attempt to make changes so it will run in vLLM.
Method
Quantised using casper-hansen/AutoAWQ and the following configs:
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
Inference
The quantised model's configs and weights are stored in hf and safetensors format, but the tokeniser remains in mistral format. Please load inference arguments accordingly, e.g.,:
vllm
vllm serve cpatonn/Devstral-Small-2507-AWQ --tokenizer_mode mistral --config_format hf --load_format safetensors --tool-call-parser mistral --enable-auto-tool-choice
- Downloads last month
- 64
Model tree for btbtyler09/Devstral-Small-2507-AWQ
Base model
mistralai/Mistral-Small-3.1-24B-Base-2503 Finetuned
mistralai/Devstral-Small-2507
Install from pip and serve model
# Install vLLM from pip: pip install vllm# Start the vLLM server: vllm serve "btbtyler09/Devstral-Small-2507-AWQ"# Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "btbtyler09/Devstral-Small-2507-AWQ", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'