This model is quantized from google/gemma-4-12B-it-qat-q4_0-unquantized using the command python -m mlx_vlm convert --hf-path google/gemma-4-12B-it-qat-q4_0-unquantized -q --q-group-size 32 --q-bits 4 --mlx-path "./gemma-4-12B-it-qat-4bit-gs32". It uses the correct group size of 32, which corresponds to the 32 nibbles (uint4) of Q4_0.
Simply put, the default of mlx-lm and mlx-vlm is 64 group size, which does not match the QAT parameters that Google is using, since that targets llama.cpp, which uses 32 by default. This quantization fixes that at the cost of slightly inflated quant size (scale parameter storage size is effectively doubled).
- Downloads last month
- 33
Model size
2B params
Tensor type
BF16
·
U32 ·
Hardware compatibility
Log In to add your hardware
4-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for michaellin/gemma-4-12B-it-qat-4bit-gs32
Base model
google/gemma-4-12B Finetuned
google/gemma-4-12B-it