A user types a prompt and waits. What happens next is a pipeline of queueing, tokenization, matrix math, memory traffic and streaming. Every performance conversation is a conversation about where that pipeline spends time.

The path, end to end

A request passes through five stages:

1. Arrival and admission. The API layer parses the body, authenticates the caller, applies rate limits, and places the request in a queue. The queue may be empty or it may be the dominant cost.
2. Tokenization and templating. Text becomes token IDs. The chat template adds role markers and special tokens. Long system prompts and retrieved documents become hundreds or thousands of extra tokens.
3. Prefill. The model processes the entire prompt in parallel. Attention and MLP layers run over all prompt positions. The KV cache is written for every layer and every prompt token.
4. Decode. The model generates one token at a time. Each step reads the weights once (shared across the batch) and the KV cache for each active request, then samples the next token.
5. Streaming. Tokens are detokenized and sent to the client, usually over server-sent events or a WebSocket. The client renders them as they arrive.

A serving engine can overlap some of these stages. It can tokenize the next request while decoding the current batch. It can interleave prefill chunks with decode steps. It can stream a token before the next one is computed. But the underlying phases remain, and each has a different bottleneck.

The four numbers

Time to first token (TTFT) is the delay from request arrival to the first streamed token. It includes queueing, tokenization, prefill, the first decode step and network time.

Time per output token (TPOT) is the average time between output tokens after the first one. If a response has $N$ output tokens and takes $T$ seconds after the first token, then

Inter-token latency (ITL) is the time between two consecutive tokens. It is similar to TPOT but usually reported as a distribution: p50, p95, p99. TPOT can look fine while individual gaps are terrible.

End-to-end latency is the total time for the request:

The $\varepsilon$ covers network transfer, detokenization, and client rendering. It is often small, but not always.

A chat product might target 500–800 ms TTFT and 30–60 ms TPOT. A batch summarization job might not care about TTFT at all but care deeply about tokens per dollar. A coding agent streaming a long file might tolerate 80 ms TPOT but break if a tool call stalls for 30 seconds. The SLO is part of the product definition.

Prefill and decode are different machines

It helps to think of prefill and decode as two workloads sharing one set of weights.

In prefill, the model sees $T$ prompt tokens at once. Each token attends to all previous tokens, so attention work grows roughly as $T^2$, while the MLP work grows as $T$. The computation is dense and parallel. A modern GPU can keep its tensor cores busy. The limiting factor is usually raw compute, not memory bandwidth.

In decode, the model sees one new token per request. It must read all the weights and the entire KV cache to produce that token. The arithmetic is tiny relative to the bytes moved. A single request at batch size one leaves most of the GPU idle. This is why decode is often memory-bandwidth-bound, and why batching is the central serving trick: one weight read can serve many requests.

The exact crossover depends on the model, GPU and batch size. The roofline model makes it precise.

A worked estimate

Take an 8B model in BF16 on one H100. The weights are about 16 GB. H100 HBM bandwidth is about 3.35 TB/s. If decode were perfectly bandwidth-bound and we ignored the KV cache, one token would take at least

That is an optimistic ceiling of about 208 tokens per second for a single request. Real systems see less: KV cache reads, kernel overhead, sampling, and imperfect bandwidth utilization. A well-tuned 8B server might deliver 50–100 tokens per second per stream at low batch, and far more aggregate throughput with batching.

Now add a 4,000-token prompt at 128 KiB of KV cache per token. The cache is about 512 MiB for that one request. Every decode step must read it along with the weights. That is another 0.15 ms of ideal bandwidth — small next to 4.8 ms, but it grows with context and concurrency. At 32K context and 16 concurrent requests, the cache dominates memory even though it is only a fraction of the per-step bytes.

Prefill is a different calculation. Processing 4,000 tokens requires roughly $2 \times 8\times10^9 \times 4000 \approx 6.4\times10^{13}$ FLOPs. At 35% model FLOPs utilization on an H100 (about 346 TFLOP/s effective), that is roughly 185 ms. Add queueing, tokenization and the first decode step, and TTFT lands in the hundreds of milliseconds. Long prompts are expensive before a single token is emitted.

These numbers are estimates. Their purpose is to show the shape: prompt length hits TTFT; output length and bandwidth hit TPOT; concurrency hits both through queueing and memory.

Streaming changes the experience, not the math

Streaming does not make the model faster. It changes how the same latency is perceived. A 3-second answer that starts in 400 ms and streams steadily feels responsive. A 3-second answer that appears all at once feels like a pause. This is why TTFT is often the most product-relevant metric even when total latency is unchanged.

Streaming also creates backpressure problems. If the client is slow to read, tokens buffer. If the server flushes too aggressively, it pays per-chunk overhead. If a proxy buffers responses, the user sees nothing until the end. Measure the full path, including the proxy.

Where requests actually fail

Most production incidents are not “the GPU was slow.” They are one of these:

• A long prompt arrives and blocks the batch. One 100K-token prefill can starve every decoding request. Chunked prefill and admission control exist for this reason.
• The KV cache fills. The scheduler must preempt or reject requests. TTFT spikes and throughput collapses.
• The queue is invisible. Autoscaling watches GPU utilization, which stays high while users wait. Scale on queue depth and TTFT, not just utilization.
• Averages hide the tail. p50 TTFT is 300 ms and p99 is 4 seconds. The average looks fine; a meaningful fraction of users are unhappy.
• The model is fine but the harness is not. An agent makes five sequential calls, each with a 700 ms TTFT. The user waits 3.5 seconds before the first visible action, and no single inference metric flags it.

The last one is the bridge to the harness track. A product’s latency is the sum of its loops, not the latency of one call.

The next lesson goes one level down, into the GPU: why memory bandwidth, not FLOPs, so often decides the decode rate.