Amazing quant
Still don't have much time on my hands, but from what I could tell, your iteration is super-impressive, especially for a model at that size. Great performance as well, despite me having to offload most of it to sysram, so at least for my specific case, this is easily the gold-standard for a lightweight quant. I'd say it comprehensively obsoletes regular APEX compact.
That said, I did observe the kind of cliches that I have come to expect from this architecture family -- terms like "unadulterated" and the turn of phrase "I don't just X, I Y..." getting overused. As I believe I mentioned before, without tweaking the samplers, the only effective remedy seems to be locking a few key sparse expert blocks closer to the middle layers at Q6, which seems to have a positive influence on the expressive range of these Gemmas -- anything below Q6 rounds off a few key edges apparently. Not really a compatible strat with the kind of sizes you're aiming for, or at least, I haven't found a way to both have my cake and eat it. I grin. Not trying to salt my genuine praise of your iteration, just mentioning it. In terms of long-context attention, instruction following, creativity and intelligence, at least from what I could discern with the limited time at my disposal, this is top-notch stuff.
Thank you kindly! I'm currently iterating through different approaches aimed at that size bracket, and in the llama.cpp compatible quant pool, this seems to perform well enough.
I'm still dying to try a proper 4-bit trellis quant with a literary dataset, however that has to wait until ik_llama.cpp implements proper Gemma 4 KV cache handling. Fingers crossed.
A definite improvement in quality without file size increase, while staying in llama.cpp territory, would come from quantizing the 3-bit sparse exps at IQ3_S instead of Q3_K. To stay in the realm of HarmonicMinus size, going for IQ4_XS in both Q4_K bound blocks, and Q3_K bound blocks with an incompatible column count that end up in fallback Q4_0, would also considerably improve quality.
That said, I don't have too many hardware configurations to test this on. I have a feeling overhead from the codebooks would impact TG drastically on older hardware.
There seems to be an interesting approach to gemma-ism pruning posted here. The size cost of this vector modification is negligible, unlike keeping a few exp layers in Q6_K, though I haven't tested its effect with Q3_K yet
Very curious approach, though I'm honestly fine with Gemma-isms. My concern is more about mapping and isolating the various specific blocks and compression tolerances. So, for example, every quant that I've done so far, and tested of yours, seems to suggest that GA layers want a minimum of Q5 to maintain long-context detail attention. The quants I've done with GA at Q6, and the test inferences I've ran on external hardware with uncompressed variants of the model don't diverge significantly from the Q5 mean in this specific metric, so it doesn't feel like we're sacrificing anything there. Mid-layer compression, however, is a lot more subtle in that regard; harder to measure, which is why I'm clinging to the observation that certain Q6 distributions seem to remove clichés in output. It's not about the clichés themselves, as much as about what their presence might represent, and what they might be replacing as we shrink the resolution of the model's abstraction layers. Wish I had more time on my hands to experiment with asymmetric distributions, probably won't be able to get back to that until next month.
It is my unprofessional mechanical understanding that at lower quantizations the cliche tendency occurs when the router repeatedly picks the same experts, which see a particular deeply baked mannerism (e.g. the dreaded adjective doublestack) and lose enough linguistic depth to resort to repeating that pattern further, completing the cycle. Increasing active experts per token does not affect it much, since their contribution is weighted, and the bottom 8 in a 16 exp/t configuration barely do anything.
It would be interesting to analyze activations and see whether activating particular experts correlates to seeing particular repetitive grammar patterns to test that theory, but I lack the mechanical understanding of how to approach that, or whether that data will help. In layman's eyes knowing precisely where the simplification occurs would allow protecting these particular levels.
P.S.: I'm open to cooperative theory crafting when you have the time to spare. I believe there's room for optimizing 26b a4b quants towards specific VRAM/RAM/ctx targets combining all our observations.