Lab 01 · inference engineering

Serving Workbench

A request is a stack of budgets. The prompt needs compute, the KV cache needs memory, the batch needs a schedule, and the GPU needs to be paid for. Change one budget and watch the others move.

5 viewsBrowser onlyEducational estimate

Read this first. The model uses rounded vendor specs, a simple analytical latency model and no real kernels. It is useful for comparing configurations and building intuition. It is not a benchmark, a capacity plan or a purchasing decision.

serving / workbenchLlama 3.1 8B · BF16 · H100 SXM ×1

Avg TTFT76 msSLO 800 ms

TPOT9 msSLO 60 ms

Output throughput728 tok/s2.44 req/s

In SLO18/182.44 req/s goodput

Where the milliseconds go

One request at the current workload and policy

Request journey

tokenize7 ms

queue0 ms

prefill56 ms

decode2.56 s

first paint20 ms

time spentTotal observed latency ≈ 2.64 s

TTFT ≈ tokenize + queue + prefill + first paint
E2E ≈ TTFT + (output tokens − 1) × TPOT

What dominates, and why

Decode dominates. Every output token re-reads the weights and KV cache, so the model is bandwidth-bound and the request stays open for 300 steps. Long outputs are a memory-bandwidth problem, not a prefill problem.

Queue time here is the gap between the ideal prefill time and the simulated average TTFT. At 4 req/s with 16 concurrent sequences, a slot is not always free when a request arrives.

Guided experiments

Three ways to break the default.

Write a prediction before you move a control. Then explain the result in terms of the mechanism, not just the number.

Find the memory wall.

Select Llama 3.3 70B, BF16, one H100. Open Memory. The weights alone exceed 80 GB, so the estimate is already impossible. Switch to INT4 and read the context table. How many 8K sequences fit now? What did quantization buy, and what did it not?

Break the SLO.

Return to the 8B default. Set concurrency to 128 and arrival rate to 20/s. Watch Avg TTFT and In SLO. Then raise the max batch to 32. Throughput rises, but whose latency pays for it? Check p95 in the Batching view.

Walk the roofline.

Open Roofline. Note decode arithmetic intensity and the ridge point. Increase the max batch and watch the decode point move right. Why does batching help a memory-bound kernel? Then switch to B200 and see whether the ridge moves with it.