The spec sheet says teraflops. The workload often says terabytes per second. Inference engineering starts by knowing which one your kernel is actually limited by.

Four numbers on every GPU

A data-center GPU is not a single processor. It is a grid of streaming multiprocessors (SMs), each with many arithmetic lanes, tensor cores, registers and a small pool of shared memory. Around the SMs sits HBM, the high-bandwidth memory that holds weights, activations and the KV cache.

Four numbers matter most for LLM serving:

An H100 SXM has 80 GB of HBM3, about 3.35 TB/s of bandwidth, and roughly 989 TFLOP/s of dense BF16 tensor throughput. An H200 keeps similar compute but raises HBM to 141 GB and bandwidth to about 4.8 TB/s. A B200 pushes memory to 192 GB and bandwidth to about 8 TB/s, with much higher compute. These are different balances, not just “faster cards.”

The capacity number is easy to understand: it limits what fits. The bandwidth number is the one people underestimate. Every decode step must read the model weights from HBM. At 16 GB (an 8B model in BF16) and 3.35 TB/s, the floor is about 4.8 ms per step. No amount of tensor-core power changes that floor. You can only reduce the bytes — with quantization, a smaller model, or more batch parallelism that shares one weight read across many requests.

The roofline, without the ceremony

The roofline model compares a kernel’s arithmetic intensity to the machine’s compute-to-bandwidth ratio.

Arithmetic intensity $I$ is the number of floating-point operations performed per byte moved:

A machine has a ridge point:

For an H100 in BF16, that is roughly

If a kernel has $I < I_{\text{ridge}}$, it is memory-bound: the machine can do more math than the memory can feed. If $I > I_{\text{ridge}}$, it is compute-bound: memory can keep up, but the math units are saturated.

Why decode sits on the left

Consider one decode step for a dense model with $N$ parameters in BF16, batch size $B$.

The math is roughly $2N$ FLOPs per token, so $2NB$ FLOPs for the batch. The bytes moved are roughly $2N$ for the weights (BF16) plus the KV cache reads. At small batch, the weight read dominates:

At batch size 1, the arithmetic intensity is about 1 FLOP/byte — nearly 300 times below the H100 ridge point. The GPU’s tensor cores are mostly waiting for data. This is the single most important fact in LLM inference.

At batch size 32, intensity rises to roughly 32 FLOP/byte (ignoring KV cache). Still far below the ridge, but much better than 1. At batch size 256, it approaches 256, close to the ridge. That is why large-batch decode can become compute-bound, and why “just increase the batch” eventually stops helping latency.

KV cache reads add bytes without adding much math, pushing intensity back down. Long context therefore makes decode more memory-bound, not less.

Why prefill sits on the right

Prefill processes $T$ prompt tokens at once. For a dense model, the dominant work is the MLP and attention projections: roughly $2NT$ FLOPs. The weights are read once and reused across all $T$ tokens, so the weight traffic is still about $2N$ bytes. The arithmetic intensity is roughly

A 4,000-token prompt has an intensity near 4,000 FLOP/byte — well above the ridge. Prefill is compute-bound. That is why prefill benefits from tensor cores, fused kernels and high MFU, while decode benefits from bandwidth and batching.

Attention itself complicates the picture. The $QK^\top$ and softmax–$V$ operations grow with $T^2$ in prefill, and their memory traffic depends on the kernel. FlashAttention exists largely to tile attention so it does not materialize the full $T \times T$ matrix in HBM. The roofline still applies; the constant factors change.

Utilization is not a single number

Two utilization metrics are useful:

• MFU (model FLOPs utilization) — achieved model FLOPs divided by hardware peak. Prefill might reach 30–50% on a well-tuned stack; 100% is a fantasy.
• MBU (model bandwidth utilization) — achieved bytes per second divided by peak bandwidth. Decode might reach 60–85% of peak on a good kernel.

A serving team that reports only “GPU utilization” is usually reporting a coarse sampling of whether any kernel is running. A GPU can be 100% “utilized” while moving far less data than its bandwidth allows. Measure the bottleneck, not the activity.

A concrete comparison

Suppose you serve an 8B model. On an H100, the BF16 weights are 16 GB, leaving roughly 64 GB for KV cache and workspace. On an L40S, the 48 GB card leaves about 32 GB, but its bandwidth is only about 0.86 TB/s. The L40S has plenty of compute for prefill but will decode slowly because each token requires the same 16 GB read over a much narrower memory pipe.

Now suppose you serve a 70B model in BF16. It needs about 141 GB just for weights — more than one H100 or H200. You need at least two H200s, or four H100s, or quantization. Once you split the model, the interconnect becomes part of the roofline: tensor parallelism adds an all-reduce on every layer, and PCIe is an order of magnitude slower than NVLink. The “compute” number on the spec sheet no longer describes the system.

This is why capacity planning starts with the model and the workload, not the GPU. The GPU is a constraint solver: capacity, bandwidth, interconnect, price.

What to measure on a real system

For a given model and engine, sweep batch size and sequence length, and record:

• tokens per second per request (decode rate),
• aggregate output tokens per second,
• TTFT and TPOT at p50/p95/p99,
• HBM used and KV-cache occupancy,
• achieved memory bandwidth and MFU if the engine exposes them.

Plot tokens per second against batch size. You will usually see a steep initial rise as weights are amortized, then a plateau as compute or KV memory becomes the limit. The knee of that curve is the operating region worth understanding.

Next: the transformer itself, seen through a serving lens. Which layers dominate memory, and why the KV cache is the real second model.