A GPU is most efficient when it has a lot of work to do in parallel. A user is happiest when their request is the only one. A scheduler is the negotiation between those two facts.

The basic trade

At batch size 1, each decode step reads the full weight matrix to produce one token. At batch size 32, the same read produces 32 tokens. The weights are amortized; throughput rises sharply. This is why serving systems batch.

But batching has costs:

• Requests wait for other requests to arrive or finish.
• A long request can hold a slot while short requests queue behind it.
• Larger batches increase per-step latency and KV-cache pressure.
• A batch that is “efficient” on average can have a terrible tail.

The scheduler’s job is to choose a batch that maximizes useful work subject to latency targets and memory limits. “Maximize batch size” is not a policy. It is a way to avoid writing one.

Static batching

Static batching is the simplest approach:

1. Collect a fixed number of requests, or wait until a batch is full.
2. Run prefill for all of them.
3. Decode until every request in the batch finishes.
4. Start the next batch.

Static batching is predictable and easy to implement. It is also wasteful. If one request generates 2,000 tokens and the other 31 generate 100, the GPU keeps decoding the long request while 31 slots sit idle. New arrivals wait for the entire batch to drain.

For offline batch jobs where latency does not matter, this is fine. For interactive serving, it is usually the wrong default.

Continuous batching

Continuous batching, also called iteration-level batching, changes the unit of scheduling from “a batch” to “one decode step”:

• At each step, the scheduler chooses which active requests to include.
• When a request finishes, its slot is freed immediately.
• A waiting request can join the next step without waiting for the whole batch to finish.
• Prefill can be interleaved with decode, or chunked to avoid blocking.

This keeps the GPU busy under bursty traffic and dramatically improves throughput for mixed-length workloads. It is a core feature of vLLM, TensorRT-LLM, SGLang and TGI.

Continuous batching is not a free lunch. It introduces:

• Per-step scheduling overhead. The engine must build attention metadata for a changing set of sequences.
• Fairness questions. Which waiting request gets the next slot? FIFO is simple but can starve long prompts.
• Latency variance. A request may join a batch with very different sequence lengths, changing its TPOT.
• Memory pressure. Admitting more requests increases KV-cache occupancy and the risk of preemption.

The latency metrics that matter

A serving SLO is usually a pair of constraints:

• TTFT: “95% of requests start streaming within 800 ms.”
• TPOT or ITL: “95% of inter-token gaps are under 60 ms.”

Plus an availability or error-rate target. These are not interchangeable. A system can have great TTFT and terrible TPOT, or vice versa.

Goodput is the throughput that meets the SLO:

A benchmark that reports 10,000 tokens per second while half the requests miss their latency target is not serving 10,000 useful tokens per second. It is generating tokens that arrive too late. Capacity planning should target goodput, not raw throughput.

Queueing, intuitively

Even a fast server has a queue when arrivals outpace service. Queueing theory gives a useful warning: as utilization $\rho$ approaches 1, waiting time grows nonlinearly. For a simple M/M/1 model, average time in system is

where $\lambda$ is arrival rate and $\mu$ is service rate. At 50% utilization, the wait is $2/\mu$. At 90%, it is $10/\mu$. At 99%, it is $100/\mu$.

Real serving systems are not M/M/1 — service times vary, batching changes $\mu$, and admission control shapes $\lambda$ — but the shape is right. Do not plan to run at 95% utilization if you care about tail latency. Leave headroom for bursts, long prompts and the occasional slow kernel.

Little’s Law is also useful:

where $L$ is the average number of requests in the system, $\lambda$ is throughput, and $W$ is average time in system. If you want 20 requests per second and each takes 2 seconds, you need about 40 requests of concurrency. If your KV cache can only hold 30, you cannot meet the target without reducing latency or adding capacity.

Choosing a batch policy

There is no universal best policy. A useful starting point:

The right max batch size is an empirical question. Sweep it. At small batch, throughput rises fast. At some point, TPOT and memory pressure make additional requests wait longer than they are worth. The knee is your operating point, not the maximum the engine allows.

A common mistake: optimizing the average

Suppose your p50 TTFT is 300 ms and your p99 is 4 seconds. You “improve” the system by reducing p50 to 250 ms, but p99 stays at 4 seconds. Most users are happier on average; the unluckiest users still have a terrible experience, and they are often your most valuable customers (long prompts, complex tasks, peak hours).

Look at the distribution. Segment by prompt length, route, tenant and time of day. A p99 that only appears for 100K-token prompts is a different problem from a p99 that appears everywhere.

The scheduler as a product decision

Every scheduling policy encodes a value judgment:

• FIFO says arrival order matters most.
• Priority says some users or requests matter more.
• Shortest-job-first says latency for short work matters most.
• Fair queueing says no tenant should monopolize capacity.
• SLO-aware scheduling says meeting deadlines matters more than order.

There is no neutral choice. Write down the policy, measure it, and revisit it when the workload changes.

Next: the memory manager behind the scheduler — paged allocation, prefix reuse, preemption and the policies that keep a serving engine alive under pressure.