The model outputs a score for every token in the vocabulary. Turning that score into the next token is a small decision with large product consequences — and a large performance surface.

From logits to a token

At each decode step, the model produces logits $z \in \mathbb{R}^{V}$, where $V$ is the vocabulary size. Softmax turns them into probabilities:

where $T$ is the temperature. Low temperature sharpens the distribution; high temperature flattens it. Temperature does not add knowledge; it changes how adventurous the sampler is.

Then a decoding strategy picks a token:

• Greedy — take the argmax. Deterministic in principle, but floating-point non-associativity and batching can still change results.
• Top-k — keep the $k$ highest-probability tokens, renormalize, sample.
• Top-p (nucleus) — keep the smallest set whose cumulative probability exceeds $p$, renormalize, sample.
• Min-p — keep tokens whose probability is at least a fraction of the top token.
• Repetition penalty — downweight tokens that already appeared.
• Beam search — keep multiple partial sequences and expand them. More compute, often less natural for chat.

Most chat products use temperature plus top-p. A temperature of 0 is often implemented as greedy, but “deterministic” is a claim about the system, not just the sampler. Batching, kernel nondeterminism and model updates can change outputs.

Sampling is part of the contract

Sampling settings affect safety, factuality, formatting and reproducibility:

• High temperature increases diversity but also the chance of incoherence or policy violations.
• Top-p too low can produce repetitive or brittle output.
• Top-p too high reintroduces the long tail you were trying to remove.
• Repetition penalties can fix loops but also suppress legitimately repeated tokens (code, lists, names).
• Seeds are not a portability guarantee across model versions or hardware.

If reproducibility matters, pin the model version, the tokenizer, the sampling parameters, the prompt template and as much of the serving stack as possible. Record the seed, but do not assume it is sufficient.

Constrained decoding

Sometimes the next token must satisfy a grammar or schema. A JSON mode, a SQL dialect, a fixed set of labels, a tool-call schema — these are constraints on the output space, not suggestions.

Constrained decoding masks out invalid tokens at each step, so the model can only choose from allowed continuations. Implementations range from simple prefix matching to full grammar-based parsers. The benefits are obvious: valid JSON, fewer retries, safer tool calls. The costs are less obvious:

• Kernel overhead for building and applying masks.
• Quality tradeoffs when the grammar forces the model away from its preferred continuation.
• Schema design matters. A grammar can guarantee syntax but not semantics.
• Latency can increase, especially with complex grammars.

A schema-constrained tool call is still only as good as the tool schema. If the schema allows a destructive action with a vague parameter, valid JSON can still be dangerous.

Speculative decoding

Decode is memory-bound at small batch: each step reads the weights to produce one token. What if a cheaper model could guess several tokens, and the target model could verify them in one pass?

That is speculative decoding:

1. A draft model (or draft heads) proposes $k$ tokens.
2. The target model runs once over the prompt plus the draft tokens, producing probabilities for each position.
3. A verification rule accepts a prefix of the draft tokens and rejects the first one that does not match the target distribution.
4. The target model’s own token at the rejection point is used, and the process repeats.

The key theoretical property: with the right rejection sampling rule, the final output distribution is exactly the target model’s distribution. Speculation is a latency optimization, not a quality tradeoff. If an implementation simply accepts draft tokens without verification, that property is lost.

The speedup math

Let:

• $k$ = number of draft tokens per round,
• $\alpha$ = acceptance rate (probability a draft token is accepted),
• $c$ = cost of drafting $k$ tokens relative to one target step,
• $v$ = cost of verification relative to one target step.

A simplified expected number of accepted tokens per round is

for a geometric acceptance model, capped at $k+1$ tokens including the target’s correction token. The effective speedup is roughly

If $\alpha = 0.8$, $k = 4$, the expected accepted tokens are about 3.36. If drafting costs 0.2 target steps and verification costs 1.0, the speedup is about $3.36 / 1.2 \approx 2.8\times$. If acceptance falls to 0.4, the expected accepted tokens drop to about 1.6, and the speedup falls below 1.5×. If the draft is expensive, the whole thing can be a net loss.

This is why speculation wins on:

• Low batch sizes, where the target model has spare compute per step.
• Predictable output, where the draft agrees often (code completion, summarization, structured text).
• A cheap, well-aligned draft model or draft heads.

It loses on:

• High batch sizes, where the GPU is already saturated and there is little spare compute.
• Highly diverse or creative output, where acceptance is low.
• A draft model that is almost as expensive as the target.

Draft architectures

• A smaller model from the same family — easy to align, but adds memory and serving complexity.
• Medusa heads — extra decoding heads on the target model, trained to predict multiple future tokens. No separate model, but added parameters and training.
• EAGLE — a draft head that operates on features rather than tokens, often improving acceptance.
• Lookahead decoding — uses the target model itself to generate candidates without a separate draft model.
• Prompt lookup / n-gram — cheap drafting from the prompt or recent context. Surprisingly effective for code and retrieval-heavy tasks.

Each approach trades acceptance rate, memory, implementation complexity and training cost. None is universally best.

The systems view

Speculative decoding is not just a model trick. It changes the serving engine:

• The scheduler must account for variable numbers of accepted tokens per step.
• KV cache allocation must handle draft tokens that may be rejected.
• The verification pass has a different compute shape than decode.
• Metrics must separate draft cost, acceptance rate and target pass cost.
• Batching interacts: a batch with one high-acceptance request and one low-acceptance request may not benefit uniformly.

A common production pattern is to enable speculation only for certain routes or batch sizes, and to monitor acceptance rate continuously. A drop in acceptance can silently erase the speedup.

What to measure

• Acceptance rate per request and per route.
• Accepted tokens per target pass.
• TTFT and TPOT with and without speculation, at realistic batch sizes.
• Quality on your eval set. Even distribution-preserving speculation can change behavior if the implementation is wrong.
• Cost per completed task, including draft compute.

Next: parallelism and MoE — what happens when the model does not fit, and why the network becomes part of the roofline.