Voxtral-Small-24B-2507-GGUF

GGUF builds of mistralai/Voxtral-Small-24B-2507 for voxtral.cpp, a ggml-based C++ runtime (Metal / CUDA / Vulkan / CPU).

The offline Voxtral model is a Whisper-style audio encoder + Mistral-Small-24B decoder that transcribes by generating text tokens. Higher quality than the 3B Mini, at a larger size.

Quants

File	Approx size	Notes
Q6_K	~20 GB	highest quality here
Q5_K	~17 GB	high quality
Q4_K_M	~15 GB	recommended — best size/quality trade-off
Q4_0	~14 GB	smaller
Q3_K	~11 GB	smallest, some quality loss

Usage

# build voxtral.cpp, then:
./build/voxtral --model Q4_K_M.gguf --audio clip.wav --gpu metal

16 kHz mono WAV; long audio is transcribed in 30 s windows. GPU auto-detected (--gpu metal|cuda|vulkan|none). Needs a GPU/RAM budget of ~15 GB+ for Q4_K_M.

License

Apache 2.0, inherited from the base model.