mradermacher

/

gemma-4-31B-it-qat-q4_0-unquantized-heretic-i1-GGUF

like 5

So just to preface this, I've redownloaded IQ3_XX3 twice and I've verified that the sha256 hash matches that in the repo, so it's not the file having been downloaded incorrectly.

I tried running IQ3_XXS over in koboldcpp 1.114.1, and I'm getting incoherent channel spam like this:
<|channel>
<|channel>thought<|channel><|channel><channel|> rest<|channel>thought<|channel>
thought
<|channel>

<|channel>thought<|channel>
<channel|>// samethoughtthoughtthought**<|channel>
thought
<channel|>thought

<channel|><|channel><|channel>thoughtthought<channel|>
<channel|>thought<|channel>thought<|channel>thought<|channel>thoughtthoughtthought
<channel|><|channel>thought<|channel>
<|channel><|channel> umathoughtas own<channel|><|channel> same<|channel>
thoughtthought<|channel>

The channel spam seems to happen regardless of if it's at the beginning of a conversation or midway through one;

I went and downloaded the IQ2_M quant (and I checked, the sha256 hash matches), and got this sorta response when given the prompt 'Write a short story about mice' (when I dropped the temperature all the way down it decided to just repeat a variation of 'NorthNorthmultMultmultNorth'):
kumb),\ persp instantiationorigಕಾಶNorth блаFanNorthNorthmultPatOctmult disput бла अर्Oct instantiationണ്ണPat ShabdNorthNorthmultƠ pattMultmultorigNorthSar бла drowsinessണ്ണ अर्носитmult persp두ണ്ണ duraMultकायाtaitOct💙multOct блаmultmultOctsalaOct graduNorthMult पिट patt甜 небо parenchymaorigNorthELA бла бланоситhausen甜multSar gradumultMultglyphorigNorthNorthNorth бла Sala इसकेmultELAноситноситmultносит sempNorthNorth💙💙Octgosноситmultipat甜淯Mult DamsNorth두multNorthorigورتmult पिटMultಕಾಶ पिटಕಾಶ patt DisputeNorthNorthMultMultoxel perspноситOctNorthNorthOct perspOct disputmult gradu instantiation두носитPat instantiation pattNorth disputноситOctNorth disputNorth disput блаमतिMult Dispute бла kumbOctmultNorthMultносит甜 SarajevoMultNorth пока disputноситipatOct gradu блаmult perspOct甜hausenmultMultNorth Shabdноситmultpettolstm),\multноситMultmultmultOctNorthNorthipatELAhausenносит HollyMultMultണ്ണ semp блаFan!} бланосит блаmult блаipatноситmultNorthNorthmultಕಾಶಕಾಶ блаMultNorth),\North图中 блаNorthноситNorthносит блаorigmult Shabd бла perspNorthಕಾಶ)

I did not have these problems when using a quant higher than IQ3_XX3. I downloaded both IQ3_XS and IQ3_M and both of those models generate coherent output (ie, an actual story about a mouse). IQ3_XXS worked perfectly fine in your gemma-4-31b-it-heretic-ara-i1-GGUF (granted, that one is non-QAT), so it's not that quant-level specifically, but maybe just a result of QAT perhaps making it more sensitive to other quants? Not sure beyond unqualified speculation on my part, but it seems that specifically IQ3_XXS and IQ2_M in this repo are incoherent / 'broken'. I haven't tried any others beyond the four I've mentioned (IQ2_M, IQ3_XXS, IQ3_XS (works), and IQ3_M (works));

While I'm here-- I'd also like to take a moment to thank you (mradermacher) for the sheer quantity of quants you've provided for literally tens-of-thousands of models