DavidAU

/

Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-NEO-CODE-Di-IMatrix-MAX-GGUF

like 685

I would also like an MTP variant as well. As a stopgap measure, I've optimized my ngram settings which gave a ~50% speed boost.

    --spec-type ngram-mod \
    --spec-ngram-mod-n-match 24 \
    --spec-ngram-mod-n-min 24 \
    --spec-ngram-mod-n-max 86

I will try it out but PiehSoft grafted MTP heads to this model and I am getting better speeds, check the finetunes or quantization links. I have to say this model is a whole other level, even the q4 quant. I am using the q6 with q8 full kv on an RTX PRO 5000 and have not seen any loops yet even mid way the full context. It has been reasoning quite well but does not seem to over think. Doing quality work for me right now. Does ngram-mod work together with mtp ok? Any chance for it to degrade quality? Have not used it yet.

If there was a way to make a version that will fit a 16GB card for my hermes agent that would be great!

You have to choose between ngram and MTP, they cannot be combined. A well trained MTP layer will always perform better, so it's the preferred option when available.

Any type of speculative decoding (ngram, MTP) will not affect output quality. The only negative thing it can do is make your responses slower if configured wrong.

According to Gemini slop they can work together and empirically the speed looks better but I will test with a few prompts and clean context and see.

Oh gemini might be right. Just realized llama.cpp supports a comma separated list for spec-type.

@markgb1 @bissli I think you boys will be happy to hear what I found.

Ran the controlled test on the ngram-vs-MTP-vs-combined question since it's this model's MTP graft we're talking about (I'm the PiehSoft side of that). And bissli was right, they compose, and the gain is bigger than expected.

Setup: Qwen3.6-40B-Deckard-MTP Q6_K, RTX Pro 6000 Blackwell, llama.cpp, q8 KV, two task types, an "edit" prompt where the output echoes the input (rewrite an existing function) and a "novel" prompt (write fresh prose). Decode tok/s, warm cache, 3 trials averaged:

config        edit (echoes input)   novel (fresh text)
mtp-only          80.5 t/s             58.0 t/s
ngram-only       108.3 t/s             40.3 t/s
combined         157.6 t/s  (200+      57.0 t/s
                             steady-state warm)

Combined config:

--spec-type draft-mtp,ngram-mod \
--spec-draft-n-max 2 \
--spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 24 --spec-ngram-mod-n-max 86

The key finding: combined BEATS both parents on editing and TIES the better parent on novel gen. It's genuine composition, not a fallback picking one. The server stats show both ngram-mod and draft-mtp firing per request with separate acceptance counts, they cover complementary token types. ngram catches the echoed tokens (variable names, unchanged lines, the function signature), MTP catches the novel ones (the actual logic changes). On editing that's a huge overlap win; on novel gen ngram just whiffs cheaply (failed drafts are microseconds) while MTP carries it, so there's no penalty.

Practical upshot: combined is strictly better than MTP-only for coding agents that spend most of their time rewriting existing files. ~2x decode on edits, no downside on generation, ~2GB VRAM for the spec contexts. markgb1's ngram params (n-match 24, n-max 86) are a solid starting point, I haven't tuned them for my workload yet, there may be more.

No quality change either way, spec decoding is lossless by construction, the draft only proposes and the target verifies. Confirmed no looping or quality drift across the runs.

@markgb1 @bissli I think you boys will be happy to hear what I found.

where is the MTP version? or the MTP heads that i can use to graft onto the 40b? i tried to graft the 27b heads but of course it didn't work.

@markgb1 @bissli I think you boys will be happy to hear what I found.

where is the MTP version? or the MTP heads that i can use to graft onto the 40b? i tried to graft the 27b heads but of course it didn't work.

I was very successful on my own. If you go to my repo you’ll see my model card. PiehSoft/Qwen3.6-40B-Deckard-MTP

My ngram suggestions were optimized for ngram-only usage. Being able to mix and match is a very new addition to llama.cpp that was added within the last month.

Here's my initial thoughts: When you're combining Ngram+MTP it likely makes more sense to minimize ngram's role, rather than wasting extra compute on ngram when MTP could have done it more accurately. I would suspect the configuration below to result in a higher sustained average speed in real-world usage.

    # NOTE: The order in `spec_type` is not respected within the current  llama.cpp implementation.
    # Right now, it always tries the ngram-based drafter first and only falls back
    # to MTP if ngram has nothing to suggest. My suggestion is based on that behavior,
    # and would need to be revised in the future if llama decides to respect drafting order.
    # It's likely that faster speeds could have been achieved if we could assign our own order of operations.
    spec_type: "draft-mtp,ngram-mod"

    # Let MTP suggest up to 3 tokens at a time. MTP is very cheap to run on small
    # models, so may as well give llama.cpp the choice to let it run.
    spec_draft_n_max: 3
    # Allow MTP to give up immediately (0 tokens) when it has no useful guess.
    spec_draft_n_min: 0
    # MTP stops guessing as soon as the next token's confidence drops below this percentage.
    # 40% is a balanced value. Allows MTP to stop early if the model is genuinely unsure about the output.
    # A value between 0.2 - 0.5 is likely a good range.
    spec_draft_p_min: 0.40

    # Upper limit on how long an ngram-mod suggestion can be. In practice the length
    # is decided by how many repeated patterns exist in the recent text, so this is
    # mostly a safety cap. Ngram will rarely get close to this unless
    # the output is EXTREMELY repetitive (boilerplate, repeated code, etc.).
    spec_ngram_mod_n_max: 32
    # How many recent tokens ngram looks at when trying to predict the next one.
    # Shorter values trigger more often but with lower accuracy; longer values
    # trigger less often but are more accurate. The optimal value within ngram+MTP config
    # is likely to exist between 16 and 48, heavily dependent on your use case.
    spec_ngram_mod_n_match: 24
    # How many consecutive pattern matches ngram must find.
    # A value of 0 means "no minimum threshold", allowing llama.cpp to immediately
    # leverage MTP whenever ngram finds no match.
    spec_ngram_mod_n_min: 0

@markgb1

Hey Mark, I appreciate your input, and I've been wanting to experiment with this because I had some preconceived notions that MTP-3 generally wouldn't work in this case due to this specific model with this specific draft head. It would seem that theory was correct, at least on my hardware.

Ran both configs head to head, same -c 270000, q8 KV both, ub2048, FA on, 4 trials per task with the cold trial dropped. Decode tok/s:

                edit_echo    novel_gen
my current        205.9        59.6
your nmax3-pmin    154.6        43.1
                  -25%         -28%

Your config came out slower on both for me, by about a quarter to a third. One thing I noticed: under your config the edit_echo trials were erratic (111, 155, 197) where mine sit rock steady at 203 to 208 every run. That instability felt like a tell.

My read on why, and it lines up with the preconception: this model's grafted MTP head seems to produce useful drafts even when its own confidence reads below 0.40, so spec_draft_p_min 0.40 ends up pruning drafts that would have been accepted. It's leaving free tokens on the table rather than saving wasted compute. The p_min gate assumes the head is calibrated such that low-confidence drafts are usually wrong, and on this particular graft that doesn't seem to hold.

The other piece is the ngram n_max drop from 86 to 32. My edit workload echoes long contiguous blocks (rewriting a whole function), so the higher cap catches them in bigger chunks. At 32 it needs more drafting rounds for the same echoed text, which is probably what's driving the edit_echo loss and the jitter.

So for this specific model it looks like n_max 2 with no p_min and a generous ngram n_max wins. Totally possible your config is optimal for a differently-calibrated MTP head, which would explain the gap. Next thing I want to try is sweeping p_min (0.0, 0.2, 0.4) while holding everything else, to see whether the right answer is no gate at all or just a lower threshold than 0.40 for this head. Will report back if I find a real peak.

Appreciate you putting the annotated config up, the comments made it easy to reason about what each knob was doing.

If you and anyone else would like, I can share my scripts I've been using to test these configurations.

Quick follow-up to my own post, because I want to correct my attribution. I initially floated the ngram n_max drop (86 to 32) as part of the cause, but I don't think that holds up on closer reading of the numbers.

The tell is that novel_gen took the bigger hit (-28%) than edit_echo (-25%). novel_gen barely exercises ngram at all since the output doesn't echo the prompt, so if the ngram cap were the real driver the damage would concentrate on edit_echo and leave novel_gen roughly flat. It didn't. That points at the MTP change, not the ngram one.

So the actual driver is MTP at n_max 3 on this specific graft, and I think it's a calibration thing, not a distribution thing. The 3rd drafted token isn't wrong because it's out of distribution. It's that the draft head's confidence and the main model's actual acceptance diverge at draft depth 3 on this head. The head is reliable to 2 tokens and gets shaky at 3. And p_min doesn't rescue it, it arguably makes it worse, because the gate reads the draft head's own confidence, which isn't the signal that predicts whether the main model will accept. So you end up pruning good drafts and keeping the depth-3 misses you actually wanted gone. n_max 2 just sidesteps the whole thing by not drafting into the depth where calibration falls apart.

Net for this model: n_max 2 is the better lever, and it's not conservatism for its own sake, it's matching draft depth to where this graft's calibration actually holds.

One thing I do want to test before drawing a hard conclusion on the ngram side: my edit_echo prompt rewrites a whole function, which is one long contiguous echo and favors a high ngram n_max. But most agentic editing in practice isn't that. Modern coding agents locate the region and emit a small targeted diff, so the echo is fragmented into short runs broken by edits rather than one big block. For that pattern the match window and n_min probably matter more than n_max, and the ideal ngram config could look quite different. I'll run a diff-style edit workload and report back, since that's closer to real use than a full rewrite.

So to be fair to your config: the ngram values you picked may well be fine, my test just happened to use the one edit shape that rewards a high cap. The piece that didn't work for this model was specifically MTP 3 plus the p_min gate.

Ahhhhh that is true, your variant is using a 27B MTP graft so I would definitely expect llama.cpp confidence levels to be thrown off by this significantly. Would also make sense that MTP accuracy deviates significantly as it predicts more tokens. I would say that your current model variant should not leverage spec_draft_p_min at all.

Yeah, exactly. The 27B head was never calibrated against the 40B's hidden states, so the confidence llama.cpp reads off it doesn't match what the 40B verifier actually accepts. p_min gates on that bad signal and prunes the wrong drafts. It'd be neat if David would let me fine tune the model with the MTP built in from stock so I could get one that matches the distribution more natively, but that's totally up to him. Or maybe it's public and I haven't seen it yet. I don't know. I'd be very interested in fine tuning a variant that has MTP.

Different topic, but since people here run this model: I tested long context on it and it holds single-needle retrieval clean out to 1M tokens with YaRN (3.85x), 100% across all depths on a single RTX Pro 6000. Had to work around a llama.cpp cap bug to get there. Full writeup, config, and scripts on my repo: https://huggingface.co/PiehSoft/Qwen3.6-40B-Deckard-MTP/discussions/3

Quick update, I did some quick n' dirty real world "perform a code review on this PR" testing. Within that scenario, and the distilled models I tested performed best with the following settings:

    spec_draft_n_max: 2
    spec_draft_n_min: 0
    spec_ngram_mod_n_match: 48
    spec_ngram_mod_n_min: 0
    spec_ngram_mod_n_max: 32

This only uses ngram when there is a "big chunk" that it can work off of. Those draft tokens are very likely to be accepted. Everything else is passed through to MTP. Gives roughly ~9% performance advantage over MTP alone. There is also a subtle benefit of this scenario: ngram-mod shares what it learns across parallel workers. So for example, you'll get a bigger boost if you have --parallel 4 and all connections are working the same content (e.g. all 4 are agents coding in Python).

Note, I removed p_min, as it seemed any confidence cutoff would cause MTP to give up too early on distilled/fine-tuned models. I would assume p_min works better with the original models the MTP layers belong to.

I'm sure a future someone will benchmark this scenario and average it across 100 runs with 100 different models, but that ain't in my scope for the week.

I was very successful on my own. If you go to my repo you’ll see my model card. PiehSoft/Qwen3.6-40B-Deckard-MTP

Interesting. i tried grafting it on my own with the 27b head but my script failed. do you happen to have an updated script or guidance on how you got that to work please?

and can you go into more detail on why you wanted to set the spec-draft-p-min to 0.40 ? i could have sworn the better recommended value for that is 0.75 ? at least i thought i recall seeing that on unsloth guidance.

and can you go into more detail on why you wanted to set the spec-draft-p-min to 0.40 ? i could have sworn the better recommended value for that is 0.75 ? at least i thought i recall seeing that on unsloth guidance.

@veldierin We explored that a bit in the other comments on this thread. Came to the conclusion that you should skip using min_p. The biggest issue seems to stem from how these confidence ratings are calculated, which doesn't appear to be that reliable of an indicator. Or perhaps it's just not reliable for some distilled models. May as well leave it a llama's default, which is to be greedy and continue drafting as long as there is a >=1% chance the draft could be accepted.

@veldierin We explored that a bit in the other comments on this thread. Came to the conclusion that you should skip using min_p. Llama's default is to be greedy and continue drafting as long as there is a >=1% chance the draft could be accepted. The biggest issue seems to stem from how these confidence ratings are calculated, which doesn't appear to be that reliable of an indicator. Or perhaps it's just not reliable for distilled models.

Interesting - i'll have to explore that a bit more on some of the models i use as daily drivers and see what my acceptance rates average at during my coding sessions. I appreciate the tips :)

I have been busy working on my projects and not testing but I will try the settings updates. Running with 242k context on 48GB vram with vision turned on. My very rough and approximate observations based on actual work are 50-60t/s around 32k full, 40-50t/s 64k - 128k, drops to 25-35t/s close to 90% of the full context. This is on an RTX PRO 5000. I have been using
--spec-type draft-mtp,ngram-mod --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 24 --spec-ngram-mod-n-max 86
--spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0.75
However, so far only looped 2 times and when context is pretty full, rarely overthinks and generally does a very good job, when it fails fixing a bug repeatedly I use codex and it gets it quick.

@veldierin I'm gonna batch this all into one.

Grafting specifically: scripts are in my repo (PiehSoft/Qwen3.6-40B-Deckard-MTP) along with the logs and tests I ran. I just did a straight byte-copy of the head tensors, none of the existing tooling handled the injection cleanly so I rolled my own. MTP always sits at the tail end of the layers, so it's easy enough to find the offset, copy the head in, and bump the tensor index.

The thing that probably killed your script: you've gotta bump the layer count in the config (97 here). llama.cpp uses that number to figure out which layer the MTP head reads from, so if you don't bump it, it either won't load or it talks to the wrong layer. That's almost certainly what got you.

And yeah, what markgb1 said on p_min, leave it off for this graft. The head still drafts good tokens even when its own confidence reads low, so the gate ends up pruning stuff the 40B would've accepted. The confidence number just isn't calibrated to this graft, don't trust it.

Now the fun part. I went and trained a native MTP head against the 40B's own distribution, base model frozen. First of three projects I've got going on this.

Since nats are kind of unintuitive: the loss runs 0 to infinity, lower means the head's distribution is hugging the 40B's more tightly. The untrained graft sits around 0.22. The trained head's settling toward a floor of ~0.07–0.08, averaging ~0.10–0.11, 183M tokens, ~3 passes, calibrated against the top-50 logits. Still about a day of training left.

Now I'll be real, I don't have the acceptance number yet. Low KL should mean high acceptance but it's not a clean linear thing, so I'm not gonna throw out a number until I've actually run the decode benchmark. Should have real n=1/n=2/n=3 acceptance in ~2 days.


What I'll guess until then: a properly calibrated head should make p_min actually useful again (unlike this graft), and should hold up deeper into the draft chain, so n=3 probably becomes worth it where the graft tapped out at 2, and honestly n=4 might be on the table if the top-distribution match holds. We'll see what the benchmark says.

Scripts we be public if anyone wants to poke at them or replicate, and the grafting script is already in there.

I only have a 5090 on a ddr4 system with 64gb system ram and a 64gb macbook m5 max, so running this specific finetune can only be done with a low context that isn't enough for my coding work (i'm still trying to figure out how to get decent dense performance on my m5 max, as it's god awful due to 600gb/sec memory bus).

i'm just enjoying learning from the discussion here. my daily driver on my 5090 is the 27b that @DavidAUmade, with the MTP heads grafted on. It's good for conversation (guiding my hermes agent), research tooling (my daily hermes jobs) to consolidate and present me information/news, and my opencode work on a project i've been coding for the last 6 months.

Ironically enough, I have a MacBook M5 Max 128GB and I'm going to be testing my newly trained MTP head. I have a specific vision for people who are not VRAM-constrained (at least on your MacBook, or unified memory) but you are bandwidth-constrained. The biggest leverage that you have is making sure that your MacBook is not sitting there idle while it waits to stream weights from VRAM.

I haven't actually tried to run the model on my MacBook yet, so I'd be curious to see what tokens per second you get. The practical limit, based off of the numbers I've calculated for a Q6 bit, is 10 to 12 tokens per second on its own. With my MTP graft that I did, theoretically, you should be in the late teens to early 20s tokens per second, which makes it a bit more usable.

If, and I mean if, my MTP head that I trained substantially increases acceptance, then the real number potentially shoots up to 18 - 27tk/s which becomes a much more pleasant experience. If you downgrade (which is basically free Q5 is ~= Q6), then it shrinks to 28GB which makes the base ~14tk/s, and now the numbers more practically shoot up to 25 - 33 tk/s.

As for your context limit, realistically, you'd have to shrink to the Q4 quant, at ~24GB, and then crush you K to Q8, or maybe even Q6, and your V to Q4. That should roughly let you fit around 262k context, as the math comes out to ~5.3 GB for full KV cache. add the overhead, and it's tight, on the size, esp if you want MTP, but you can get this model to fit.

The way I've set it up is I use my main PC as the tunnel to wire into my open code harness. One technique I found if you don't use your PC all that often is to set up an SSH tunnel through Cloudflare into your machine before Windows has the chance to do anything. The reason why is that Windows eats up roughly 2 GB of VRAM on its own, meaning you have effectively only 30 GB of VRAM to work with. Due to MTP and a couple of other factors, you might spike a little higher than your budget allows for. Probably not the full 262,000 context, but it could get you close enough.

When you SSH into the machine, you get to claim that VRAM before anything else does on startup. Neat trick that I figured out when I was trying to use up my 3090 as a local agent.

I've been playing around with optimizing Pieh's grafts. My 'quick' MTP re-training attempt via top-50 did not change acceptance rates or confidence. Here's a simplification of the most important steps I took:

• NON-TECHNICAL: At every spot where the model is about to predict the next word, look at the big model's full list of guesses (all 248,000 possible words). Save its top 50 favorites along with how confident it was about each one. Also save the single word it actually chose and the word that came right before it.
  • TECHNICAL JARGON: For each token position, the verifier's full 248k-vocab softmax was reduced to its top 50 indices + log-probs, plus the verifier's chosen target_token and the preceding prev_token.
• NON-TECHNICAL: Train the draft model to mimic the big model's confidence levels, but only on the same 50 favorite words (from above). Anything outside the top 50 was hidden from the MTP draft during training.
  • TECHNICAL JARGON: KL divergence between the draft MTP head's softmax and the verifier's softmax, both restricted to those same 50 indices. Everything outside top-50 was masked.

This was supposed to show that the retraining could actually nudge the needle to match the new 40B neural network layout... but ended up proving nothing unfortunately. I will be continuing to probe at solving this for a bit.

However, since Pieh mentioned he's also playing around with this, am going to share some notes MiniMax M3 provided by analyzing my results:

Summary

• The results are identical: baseline 1.74% vs trained 1.74% (43/2473). The training did not improve MTP acceptance. The donor head was already a reasonable Qwen3.6 MTP head from Brian6145, so the KL was already low (~12) and the small gap (KL 1.19) wasn't large enough to flip tokens in the acceptance test.

Why the same acceptance

• The trained head matches the verifier on KL over the top-50 vocab distribution, but the single greedy token at each position is what controls MTP acceptance. KL on top-50 can drop from 12 → 1.19 while still leaving the argmax of the draft unchanged in ~98% of positions. The two prompts that did accept (5.99%, 11.49%) — and the same 8 that didn't — are byte-identical between baseline and trained.

What would actually help

1. Train against full softmax (not top-50). The current loss is KL(soft(draft_top50) || soft(verifier_top50)) — both clipped to the same 50 indices. The argmax can stay put even when the distribution mass shifts inside those 50 slots.
2. Train with a token-level cross-entropy penalty (-log p_draft(target_token)), which directly optimizes the argmax.
3. Use a true distillation of the verifier's hidden state (e.g. feature alignment of the MTP block's last hidden layer to the verifier's first decoder-block hidden state at the same position) — that's the upstream signal the MTP head is supposed to consume.
4. More training data and longer context (192 tokens/prompt is short; the MTP head sees almost no structural signal beyond local n-gram).

@markgb1

Interesting results, and I think they actually point at something the n=1-focused framing misses. Quick counterpoint first: retraining can move acceptance, my trained head went from ~0.91 to ~0.945 at n=1 on code over a clean benchmark (thousands of drafts, not a short sample). So the "needle doesn't move" conclusion looks specific to your run, probably because your donor (Brian6145's head) was already well-matched at n=1, leaving little argmax to fix, plus 192-token prompts / 43-of-2473 is a low-signal regime where baseline and trained will read byte-identical even if the distribution shifted. I used the direct MTP head straight from Alibaba.

But here's the part I think matters more: n=1 isn't where the real problem is. I measured the full depth ladder, and the story is a cliff, the trained head is ~0.945 at n=1 but 0.66 / 0.50 / 0.38 at n=2/3/4, while the graft decays gracefully (0.85 / 0.76 / 0.68). The root cause was single-token distillation only ever trained the head to predict position 1 from the body's hidden state. At inference the head drafts depth>1 by consuming its own hidden state, which it never saw in training. Which, is classic exposure bias that I found out only after 2 days of training. Your suggestion #3 (distill the upstream hidden signal) is the closest thing in this thread to the fix, but the version that works is training the head unrolled k steps deep, feeding its own hidden state forward at each depth with KL at every position. Early runs show the deep-position losses actually coming down, which single-token training never touched.

So your top-50 -> full-softmax and CE points are reasonable for n=1 polish, but I don't think they'll fix depth, since depth needs the multi-token objective. Happy to share more once I've got the full n=1–n=4 retrained numbers; mid-retrain now.

Yep agreed - I took too many shortcuts in the interest of proving the methodology in minimal time. Mentioned above, I ended up proving nothing 🤣

Will make the pipeline more robust and continue with training and grafting onto David's Q8_0 quant.

On a tangent, all this has made me realize - this repo probably should have used Qwen FP8 as a base rather quanting down to Q8. Would have given a nice performance boost for those with native 8-bit hardware acceleration.

My attempt at MTP retraining was far more successful than I was expecting. I've managed to bump spec-draft-n-max=3 acceptance rates to ~87%. This means my new model's MTP layer provides a HUGE 3x speed boost.

As far as I can tell these acceptance rates are near the ceiling, and the missing ~13% acceptance seems to be dependent having access to the verifier's rich hidden state h_t.

The following performance test was using a real-world agentic scenario, where I am prompting within BrowserOS "find me the most popular tourist destinations" and capturing the average response speed. This was run on a single Nvidia DGX Spark with ngram disabled (pure MTP) and temperature 0.6.

Non-technical MTP training description

• Ran the big model on 117 random prompts and recorded what it was thinking at every position.
  • Specifically, recorded which 64 tokens it thought were most likely next and how confident it was in each.
• This generated enough datapoints to train the MTP layer on the new 40B layout

Technical notes

• I had substituted h_t with embedding(input_token)(5120-dim vector) that is just the lookup of one token in the embedding table.
  • That token embedding does not carry information about the prompt, the previous response, or the reasoning mode. It's effectively a bigram feature.
• This method relies on the fact that chat responses use a relatively fixed vocabulary of tokens, so the unigram prior is strong.
• Since the trained head is shown real-world verifier logprobs during training, so its bias toward the most common tokens is effectively re-calibrated.
• Notably, this model the guesser's architecture had to be in PyTorch in order to match the existing C++ implementation exactly so the weights could be grafted back.
• The results are better than I expected, likely because the MTP was already "approximately close" to matching the modified 40B layout.
• I've likely approached the information-theoretic ceiling of "predict from one token embedding."

Next steps?

The only step I can think of to improve acceptance rates would be rebuilding llama-server with a small edit to expose hidden states. This would allow the head to actually use context and likely hit >95% acceptance.

Will soon upload my model and scripts to share what I've learned/accomplished with the community.

My v1 re-trained MTP (attached to a Q8_0 base model) is listed here: https://huggingface.co/markgb1/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-iMatrix-MTP-GGUF
Currently training up a v2. See the repository readme for continued status updates.

I tried several different ways of training using data collected from the verifier's hidden states. Every version was significantly worse than v1, so was not worth the upload.

Things I've tried:

• Loss functions: CE, CE+KL+MSE, KL+CE
• Data: real h_t, logprobs, token emb proxy
• Corpus: 8, 21970, 9463 triples
• Epochs: 3, 4, 5, 10
• LRs : 5e-5, 5e-4

Will poke around a bit more but this may potentially end as a stalemate.

@WTPieh any luck on your end?

My v1 re-trained MTP (attached to a Q8_0 base model) is listed here: https://huggingface.co/markgb1/Qwen3.6-40B-Claude-4.6-Opus-Deckard-Heretic-Uncensored-Thinking-iMatrix-MTP-GGUF
Currently training up a v2. See the repository readme for continued status updates.

Any chance for a Q6 later? I like the speed and I am also using the mmproj. Will be tight on my RTX PRO 5000.

Hey everyone, sorry I've been heads-down, only just got to sit down and compile this. Short version: I tried a lot too, and I landed in the same place you did, @markgb1, the original Qwen head is effectively the best match for this model. The drift is within a couple points and doesn't really matter. These numbers are from ~3 days ago.

Controlled sweep, n=1–4, pure MTP (ngram off), all heads bf16 in a Q6_K body, code workloads, acceptance from the server's draft-mtp stats line:

The thing that convinced me it's a stalemate: I measured the native 27B head on its own 27B body as a reference, and the grafted-onto-40B numbers are within ~0.02 of it at every depth. So the 27B head's next-token geometry basically survives the expansion + abliteration + fine-tune intact — there's no drift to win back, which is why retraining matches but won't beat the graft.

One thing that might be useful for your h_t attempts though, @markgb1, the failure mode there is easy to hit: if the head is trained single-token (predict only +1 from h_t + gold token), it gets a great n=1 and then collapses at depth. I saw exactly that, a single-token head went 0.945 → 0.659 → 0.495 → 0.382.My first pass was trained on over 183M tokens. The fix is a multi-token unroll: feed the head its own hidden state forward across k depths and KL against the body's top-k per position. That's the "my retrain" column above, and it's what holds depth. Your corpora also looked small (the 8/22k/9k triples), I needed ~45M tokens / 2 passes before depth stabilized. Might be worth a single h_t + unroll run on a bigger corpus before calling it; that's the one variant I'd still expect to move.

But yeah... agreed it's likely a stalemate. Happy to share the unroll trainer + head-swap tooling if anyone wants to push it further.

Thanks for this link!
I will use yours to remove the MTP head from it and attach it to the Q6 model because that is what I am using.

After this I can make models for other people with your trained MTP head. This 40B Model is awesome and lets me do things that other models can't. It really has value but it is quite slow and speeding it up is exactly what i need!

Update; So I did remove the MTP head from @markmonger Q8_0 base model and graft it onto the Q6_K quant instead (the one that I actually use); but this "updated" MTP head is 0% faster in my test prompts vs the original one. They both yield around 22tok/sec vs 15 tok/sec on my W7900.
I can upload the Q6_K model (both the one with the original Unsloth Q8 MTP head and the "retrained" MTP head from @markmonger ) if this is useful for anyone. I can also make MTP versions of any other quant if people want it now that I have built the scripts to do it.

I uploaded the MTP's from Q6 both the original head + post trained MTP head.