Full Breakdown
Google Accelerates Gemma 4 with Multi-Token Prediction Drafters
5/7/2026, 12:19:28 PM
Accelerating Local AI: Multi-Token Prediction Drafters for Gemma 4
Google released a set of “multi-token prediction” (MTP) drafter models for its open-weight Gemma 4 family. The drafters employ speculative decoding, generating tentative tokens in parallel with the main model and discarding them if they conflict with the final prediction. By sharing the key-value cache with the primary model, the drafters reduce the number of memory transfers required for each token, allowing the system to produce output up to three times faster while preserving the same quality.
Background: Gemma 4 and Edge-AI Strategy
Gemma 4, announced in spring 2026, is Google’s open-source large language model designed for on-device or local-server deployment. It builds on the same architecture that powers Google’s Gemini AI but is tuned for execution on a single high-power AI accelerator or a consumer GPU after quantization. The release follows a shift to the permissive Apache 2.0 license, replacing the earlier custom Gemma license, and reflects Google’s push to enable developers to run powerful models without transmitting data to cloud services.
Key Figures & Groups
- Google DeepMind – research lab that developed the MTP technology.
- Maarten Grootendorst – Google engineer who presented the performance claims.
- Olivier Lacombe – Product Management Director at DeepMind, co-author of the announcement.
- Google AI hardware team – responsible for optimizing Gemma 4 for TPU clusters and consumer GPUs.
Data & Performance Gains
- Gemma 4 includes a 26-billion-parameter model (Gemma 4 26B) that can run at full precision on a single TPU accelerator.
- The MTP drafters are lightweight, with 74 million parameters in the E2B variant.
- Benchmarks on an NVIDIA RTX PRO 6000 show token-generation throughput roughly tripling when the drafter is active, with identical output quality.
- The drafter’s sparse decoding narrows token clusters, further reducing compute cycles.
Why It Matters: Cost, Accessibility, and Open-Source Competition
Faster inference directly lowers the compute cost of running large language models, a growing concern for developers scaling AI workloads. By delivering a threefold speedup without sacrificing reasoning, Google positions Gemma 4 as a cost-effective alternative to proprietary cloud APIs. The open-source licensing and ability to run on consumer hardware broaden access for researchers, startups, and hobbyists, potentially reshaping the market for locally hosted AI.
Official Statements & Responses
Google announced that the MTP drafters “increase inference efficiency of Gemma 4 by roughly threefold” and emphasized that the approach does not degrade output quality or logical reasoning. The company highlighted the drafters’ compatibility with existing Gemma 4 checkpoints and their suitability for both TPU-based servers and GPU-based desktops after quantization. DeepMind’s product team framed the technology as a “key differentiator for open models,” noting that inference speed and accuracy together drive adoption in cost-sensitive environments.
Verbatim Quotes
- “up to a 3x speedup without any degradation in output quality or reasoning logic.” — Maarten Grootendorst, Google engineer
- “For example, the drafter shares the key value cache (essentially the LLM’s active memory) so it doesn’t need to recalculate context the main model has already worked out.” — Google technical documentation
- “Gemma allows users to tinker with AI on their hardware rather than sharing all their data with a cloud AI system from Google or someone else.” — Google product announcement
- “MTP uses that time to bypass the heavy model and generate speculative tokens with the lightweight drafter.” — Google engineering blog
- “Inference speed and efficiency while maintaining accuracy is a key differentiator for open models.” — Olivier Lacombe, Product Management Director, DeepMind
