Conversion script
#2
by MJS255 - opened
Any chance you could share the conversion script for the int8 convrot model, would like to try some abliterated models
sure: int8 & nvfp4 are made with Silveroxides/convert_to_quant tool and this command:
python -m convert_to_quant
-i clip\Gemma4\gemma4_12b_bf16.safetensors
-o clip\Gemma4\gemma4_12b_int8-convrot-simple.safetensors
--int8 --convrot --simple --heur --calib-samples 32768 --comfy-quant --save-quant-metadata
--gemma4
(swap int8 convrot args with nvfp4 if you want to make it too)
Hmmm, my abliterated BF16 text encoder works fine but after quant with your script it generates gibberish. Any ideas?
https://huggingface.co/igorls/gemma-4-12B-it-heretic-v1
This model
--int8 --scaling_mode row --simple --convrot --convrot-group-size 256
--comfy_quant --save-quant-metadata --<gemma 4>
Used these instead and seems to work fine
MJS255 changed discussion status to closed