A benchmark is a measurement of a workload under conditions. If you do not define the workload, the conditions or the SLO, you have measured a demo.

Start with the workload

Before touching a load generator, write down:

• Model and precision — exact checkpoint, quantization, tokenizer and chat template.
• Prompt length distribution — not just the average. p50, p95, p99 and the maximum.
• Output length distribution — the same.
• Arrival pattern — steady, bursty, diurnal, or synthetic closed-loop.
• Concurrency — how many requests are in flight, and whether that is controlled by users or by a load generator.
• SLO — TTFT and TPOT targets at a stated percentile.
• Mix — chat, RAG, agents, batch; each may have different lengths and priorities.

A benchmark with a fixed 128-token prompt and 128-token output is a unit test, not a capacity plan. Real traffic has a long tail, and the tail is where the system breaks.

Open-loop vs closed-loop

Closed-loop load generators send a request, wait for the response, then send the next. The number of in-flight requests is bounded by the number of simulated users. This measures latency at a given concurrency, but it can hide queueing: if the server slows down, the generator slows down too.

Open-loop generators send requests at a fixed arrival rate, regardless of how fast the server responds. This exposes queueing and overload. It is closer to real traffic, where users do not politely wait for the system to recover.

Use both:

• Closed-loop to find the latency at a target concurrency.
• Open-loop to find the maximum arrival rate the system can sustain within the SLO.

A good load test records the actual arrival times and service times, so you can reconstruct queueing.

The metrics that matter

Report percentiles, not just averages. A p50 of 300 ms and a p99 of 4 s describe two different products.

Little’s Law for capacity

Little’s Law:

If you need to sustain $\lambda = 20$ requests per second and the p95 latency is $W = 2$ seconds, you need roughly $L = 40$ requests in flight. If each request holds 8K tokens of KV cache, you need enough memory for 40 × 8K tokens. If your GPU can only hold 30, you cannot meet the target at that latency.

This is a consistency check, not a full model. It tells you whether your concurrency, latency and memory assumptions fit together.

From throughput to GPUs

A simple capacity estimate:

Where GPU-seconds per request is the inverse of per-GPU request throughput. For example:

• Peak traffic: 50 requests/second.
• Each request takes 0.2 GPU-seconds at the target batch size (i.e., one GPU serves 5 requests/second).
• Target utilization: 60% to leave headroom for bursts.

Add redundancy for failures, a warm pool for cold starts, and separate capacity for prefill-heavy and decode-heavy traffic if you disaggregate. Then measure again. The formula gives a starting point, not a guarantee.

Why utilization headroom matters

At 60% utilization, a 2× burst can be absorbed by the queue without violating the SLO. At 90%, a 20% burst can double latency. GPU capacity is expensive, so teams are tempted to run hot. The result is a system that looks efficient on a dashboard and fails during the one hour that matters.

A useful rule: size for peak, not average. If peak is 3× average and you size for average at 80% utilization, you are 2.4× short at peak. Autoscaling helps only if it reacts fast enough; GPU cold starts can take minutes.

Benchmark pitfalls

• Warm-up. The first requests pay model load, kernel compilation and cache warm-up. Exclude them, but measure them separately.
• Cache state. A benchmark with a warm prefix cache is not the same as a cold start.
• Synthetic prompts. Random tokens tokenize differently from real text and may hit different kernels.
• Output length truncation. If the benchmark stops at 128 tokens, it hides the cost of long generations.
• Client bottleneck. The load generator may saturate its own CPU or network before the server does.
• Timeouts. A timed-out request is not a completed request. Count it as an error, not a success with infinite latency.
• Version drift. Record engine, model, driver and CUDA versions. A benchmark without versions is not reproducible.
• Cherry-picked batch size. The best batch size for throughput may violate the latency SLO. Report the operating point, not the maximum.

A capacity worksheet

For each route or model:

1. Request rate at peak (req/s).
2. Prompt tokens p50 / p95 / max.
3. Output tokens p50 / p95 / max.
4. TTFT and TPOT SLOs at p95.
5. Measured per-GPU throughput at the target batch and sequence mix.
6. KV memory per request at p95 context.
7. Max concurrent requests per GPU.
8. Required GPUs from throughput and from memory; take the max.
9. Add 30–50% headroom and N+1 redundancy.
10. Re-measure under a realistic load test.

The two calculations — throughput and memory — often disagree. The larger number wins. A GPU that has enough compute but no KV memory cannot serve the request.

Cost per completed task

Tokens per second is an intermediate metric. The business cares about cost per completed task:

Include retries, failed requests and idle capacity. A system with a 20% retry rate is 20% more expensive per success, even if its token price is lower. This is where inference and harness economics meet: an agent that retries three times because the first model was cheap can erase the savings.

Next: production inference — the operational reality of autoscaling, cold starts, rollouts, multi-tenancy and incidents.