Is the quantize script opensource?
#2
by wxsm - opened
Thank you!
Haven't made it available yet. It's based on this: https://github.com/wangyuze18/DeepSeek-W4AFP8-AWQ
Haven't made it available yet. It's based on this: https://github.com/wangyuze18/DeepSeek-W4AFP8-AWQ
Thank you, amazing quantization, exicted to see the scripts opensource too!
@wxsm you can simply quantize experts to int4 since original model is already quantized with importance of activation scales. This makes your quantization script much simple : no need of dataset calibration.
If you want higher accuracy score you can quantize model with expert importance score.
AWQ was an old method without taking care of experts.