How Speculative Decoding Cuts LLM Inference Latency Without Changing Outputs

What Is Speculative Decoding, and How Does It Speed Up Autoregressive Inference?

Speculative decoding is an algorithm that samples from autoregressive language models faster without changing their outputs, by computing several tokens in parallelCITE:E1. Leviathan et al. at Google Research introduced the method in a 2022 paper, framing it as a way to accelerate sampling while producing results identical to standard decodingCITE:E1.

What Is the Core Algorithmic Insight, and What Role Does the Draft Model Play?

The algorithm rests on the observation that scoring short continuations from a faster, less powerful draft model in parallel takes about as long as sampling a single token from the larger target modelCITE:E3. Chen et al. at DeepMind describe this latency parity as the mechanism that makes the approach worthwhileCITE:E3. In a related implementation, Hugging Face's assisted generation approach has the draft (assistant) model use greedy decoding to generate a set number of candidate tokens before the target model reviews themCITE:E7.

How Does the Target Model Perform Parallel Verification in a Single Forward Pass?

The target model processes the input sequence together with all draft tokens simultaneously in one forward pass, computing a probability distribution for each positionCITE:E6. This single-pass mechanism, described by NVIDIA, is what lets the target model check multiple speculative tokens at once instead of generating them one at a timeCITE:E6.

How Does Speculative Sampling Guarantee No Loss in Output Quality?

Speculative sampling pairs with a modified rejection sampling scheme that preserves the target model's output distribution within hardware numerical precisionCITE:E4. Chen et al. present this rejection step as the guarantee that any draft token accepted into the final output does not shift the probability distribution the target model would have produced on its ownCITE:E4.

How Much Speedup Does Speculative Decoding Achieve, and How Does It Compare Across Model Scales?

Leviathan et al. report a 2X-3X acceleration on T5-XXL compared with the standard T5X implementation, with identical outputsCITE:E2. Chen et al. benchmark speculative sampling on the 70-billion-parameter Chinchilla model and report a 2-2.5x decoding speedup in a distributed setup, without compromising sample quality or modifying the model itselfCITE:E5.

What This Means

Across two model families of very different scale, both the original speculative decoding paper and the DeepMind speculative sampling paper report speedups in the same 2x-to-3x band while explicitly preserving output identity or distributionCITE:E2CITE:E5. The consistency between Google Research's 2022 formulation and DeepMind's 2023 rejection-sampling refinement, later described in near-identical single-forward-pass terms by NVIDIA in 2025 and by Hugging Face's 2023 assisted-generation implementation, indicates the draft-then-verify pattern has held as a stable technique across separate teams and years rather than a one-off resultCITE:E1CITE:E3CITE:E4CITE:E6CITE:E7.