Zephel01's Heretic-NEO-CODE Benchmarks & request for MTP
Hi DavidAU
I hope you're having a great week. First off, I just want to say a massive thank you for all the work you've been putting into the Qwen3.6-27B models. I've been following your releases for a while now, and it's honestly inspiring to see how much care and craftsmanship you pour into each version. The community is lucky to have you.
I recently came across some independent benchmark results that I thought you'd be genuinely excited to see, they confirm something pretty remarkable about Heretic-NEO-CODE model.
There's a Japanese reviewer named zephel01 who runs a really thorough coding benchmark called llmbench. He tests models on real software engineering tasks, fixing actual bugs in real repositories and he doesn't just check if the code runs; he also evaluates code quality and maintainability. It's about as close to real-world coding as a benchmark gets.
The results are pretty incredible:
Your Heretic-NEO-CODE model scored 59 out of 60 (98.3%) on his toughest 60-task benchmark. Even more impressive, it got 19 out of 20 (95%) on the hardest "architect" level tasks - the ones that require repository-level understanding and complex system design reasoning. That made it the overall 60-task champion, beating out dedicated coding models.
Your Fable-Fusion-711, which I know you put a lot of work into and which has that amazing 0.711 ARC-C score, scored 54 out of 60 (90.0%) on the same test. So there's about an 8 percentage point gap between the two.
Here's the thing that Heretic-NEO-CODE wasn't even designed for coding. It's a creative/uncensored roleplay model. Yet it outperformed dedicated coding models on the most challenging architect-level tasks.
One more thing:
zephel01 tested the MTP (Multi-Token Prediction) version of your Fable-Fusion model and found it gave roughly 1.7 to 1.9 times faster inference going from about 58 tokens per second to over 100 tokens per second. That's a massive speed boost.
So here's my question, and I know I'm not alone in asking this: would you consider releasing an MTP version of Heretic-NEO-CODE?
I honestly think that would be a game changer for developers running your models locally at MTP Pace.
Please keep investing in this line of models for coding. The benchmarks speak for themselves , you've stumbled across something really special here, and it deserves more attention.
Thank you again for everything you do for this community. Looking forward to seeing what you come up with next!
Link:
zephel01's benchmark article:
https://note.com/zephel01/n/n6e401304c118
https://note.com/zephel01/n/n36dbe4f485fa
https://note.com/ai_driven/n/n0e6a27b2a907
Quick note:
Thank you for detailed feedback and notes - these are invaluable to tuning the pipeline and models.
; I will get back to you on other noted issues.
interesting... Hey David, thanks a lot for this work and for sharing the weights.
I finally took 3.6 Fable-Fusion and 3.8 Cold-Fusion for a real run in my local nanobot + LM Studio setup today. Honestly, as a backend for that kind of workflow, they don’t feel on the same level as this Heretic-Finetune-Neo-Code. It just feels sharper and more reliable, and despite not using MTP it’s noticeably faster at actually getting things done. so MTP support would be nice ;)
I tested them on writing longer research-style stock reports. Fable started hallucinating in places; part of it was a Jinja template mismatch on my end, but it still made me nervous. Both models were also painfully slow and repeatedly hit timeouts or stalled halfway through when trying to generate/convert the PDF. Neo-Code needs 10 min for research and writing. The others between 20 and 60 min. Fable has good quality but was often mixing up which report he was doing. Cold-Fusion got stuck after one hour and couldn't recover his work. Neo-Code found the logs and got it done after verifying the data.
Again, really appreciate what you’re doing here. Just wanted to share, and second hesham165, how awesome Heretic-NEO-CODE works.
Thank you for your detailed notes and feedback ; "Finetune" is a much simpler; but more direct tune of Qwen 3.6.
It is also more generalized.
Interestingly it can be "folded into" FF711 and/or Qwen 3.8 version[s] ; something I will try at a later date and/or replicate the tune itself directly.
Should also note one other thing:
Di-matrix contains NEO and a CODER imatrix dataset combined ; this method (the model was imatrix'ed twice, then these two "averaged together") does have a slight effect on each quant.
This can be measured in PPL, KLD and 3 other metrics.
These metrics / quality interact especially at longer context and in some cases "detail work" - the newest, and higher context models especially with part/full attn tensors react strongly to this.
It would be interesting to create NEO Coder DI-MATRIX quants for Cold Fusion and Fable Fusion 711 to compare.
Also; as noted Jinja template (and even minor errors) can completely throw off both generation AND tuning.
Great!!!
Looking forward to see NEO Coder DI-MATRIX quants for Qwen3.8 and Fable Fusion 711. It's gonna be a banger!!