How do we use MTP for speculative decoding?
#6
by pythiccoder - opened
In the docs it says
Multi-Token Prediction: All Gemma 4 models (E2B, E4B, 12B, 31B, and 26B A4B) include a dedicated draft model for speculative decoding, enabling significantly faster inference with no quality loss.
how do we use this draft model in practice? Were these heads removed from the mobile quants?
pythiccoder changed discussion title from How do we use MTP? to How do we use MTP for speculative decoding?