A transformer layer looks like a handful of equations. On a GPU it becomes dozens of kernel launches, each with its own memory reads, writes and synchronization. The gap between the math and the machine is where performance lives.

The memory hierarchy

A GPU kernel does not read directly from HBM for every operation. It moves data through a hierarchy:

• HBM — large, high bandwidth, high latency. Holds weights, activations and the KV cache.
• L2 cache — shared across SMs, smaller, faster than HBM.
• Shared memory / SRAM — per-SM, very fast, small. Used for tiles and reductions.
• Registers — per-thread, fastest, tiny.

A well-written kernel tiles its work so that data is reused in shared memory or registers before it goes back to HBM. A naive kernel materializes intermediate tensors in HBM, paying bandwidth for every step. At decode time, where there is little math per byte, this is fatal.

Fusion

Kernel fusion combines multiple operations into one kernel so intermediates stay on-chip. A simple example: instead of

1. compute $xW_1$,
2. add bias,
3. apply activation,
4. write the result to HBM,
5. read it back for $xW_2$,

a fused MLP kernel can keep the intermediate in registers or shared memory and write only the final result. The FLOPs are the same; the bytes moved are much lower.

Fusion matters most when the workload is memory-bound. Decode is exactly that. Prefill also benefits, but it has more arithmetic to hide the traffic.

Common fusions in LLM serving:

• QKV projection in one GEMM instead of three.
• Bias + activation fused into the GEMM epilogue.
• Residual + normalization fused.
• Rotary position embedding fused into attention.
• Softmax + value multiply in FlashAttention.
• Sampling fused with the LM head where possible.

Each fusion is model- and hardware-specific. A kernel that is fast on an H100 may be mediocre on an MI300X, and a kernel optimized for BF16 may not support FP8. This is why serving engines ship whole kernel libraries, not a single generic path.

FlashAttention and the IO problem

Standard attention computes

If you materialize $S$ and $P$ in HBM, a long sequence creates a huge intermediate. For a 32K-token sequence with 32 heads, that is billions of elements per layer. The memory traffic dwarfs the arithmetic.

FlashAttention tiles the computation. It loads blocks of $Q$, $K$ and $V$ into SRAM, computes partial attention, and accumulates the output without ever writing the full matrix to HBM. The result is mathematically exact (not an approximation), but the memory traffic is dramatically lower.

The lesson generalizes: the roofline is about bytes, not just FLOPs. A kernel can be “mathematically efficient” and still be slow because it moves too much data. This is why IO-aware kernel design is a core inference skill.

CUDA graphs

A decode step may launch hundreds of small kernels. Each launch has CPU overhead — a few microseconds, sometimes more. If a decode step takes 10 ms of GPU work but the CPU takes 12 ms to launch the kernels, the GPU idles. The model is not slow; the launch path is.

CUDA graphs capture a sequence of kernel launches and replay them as a single graph. The CPU submits one graph launch; the GPU executes the captured operations with much lower per-kernel overhead. This is especially valuable for decode, where each step has the same shape and many tiny operations.

Limitations:

• The graph must be captured for a specific set of shapes and memory addresses. Dynamic batch sizes and sequence lengths complicate capture.
• Some operations (dynamic control flow, certain memory allocations) are not graph-safe.
• Graph replay is fast but inflexible. Engines often keep a set of graphs for common batch sizes and fall back for unusual shapes.

CUDA graphs are one reason a production engine can be much faster than a naive PyTorch loop even with the same model and GPU. The math is identical; the launch overhead is not.

The decode step, kernel by kernel

A simplified decode step for one layer:

1. RMSNorm (read hidden state, write normalized).
2. QKV projection (GEMM).
3. Rotary embedding on Q and K.
4. Append K, V to the cache.
5. Attention: read Q, the cache, compute scores, softmax, mix V.
6. Output projection (GEMM).
7. Residual add.
8. RMSNorm.
9. MLP up/gate (GEMM).
10. Activation.
11. MLP down (GEMM).
12. Residual add.

Multiply by the number of layers. For a 32-layer model, that is hundreds of kernels. Each one has launch overhead, memory reads and writes. At batch size 1, many of these kernels are tiny — a matrix-vector product with low parallelism. The GPU is underfed.

Batching helps because it turns matrix-vector products into matrix-matrix products with more parallelism. Fusion helps because it removes intermediate traffic. Graphs help because they remove launch overhead. Quantization helps because it reduces weight bytes. Each attacks a different part of the same problem.

Prefill kernels are a different shape

Prefill has large matrices: $T$ tokens by $d$ hidden dimension. GEMMs are efficient, tensor cores are well utilized, and the kernel launch overhead is amortized over much more work. The main challenges are:

• Attention tiling at long sequence lengths (FlashAttention).
• Causal masking without wasting half the computation.
• Variable sequence lengths in a batch, which require packing or padding.
• Chunked prefill to avoid blocking decode.

A prefill-optimized kernel may be a poor decode kernel, and vice versa. Engines often dispatch different kernels by phase. Some go further and disaggregate prefill and decode onto separate hardware pools.

What to profile

Do not guess. Use a profiler (Nsight Systems, Nsight Compute, ROCm profiler, engine-specific tracing) and look for:

• GPU idle gaps between kernels — launch overhead or CPU bottleneck.
• Memory throughput versus peak — are you bandwidth-bound?
• SM occupancy — are there enough warps to hide latency?
• Kernel duration distribution — a few long kernels vs. many tiny ones.
• PCIe/NVLink traffic — is communication stealing time?
• Cache hit rates — L2 and KV-cache reuse.

A common pattern: the timeline shows the GPU idle for 30% of the step, and the fix is CUDA graphs or a fused kernel, not a faster GPU.

The build-vs-buy question

You can write custom kernels, use a serving engine’s kernels, or rely on a compiler (TorchInductor, TensorRT, XLA). The tradeoffs:

• Custom kernels can be excellent for a specific model and GPU, but cost engineering time and break with model changes.
• Engine kernels are battle-tested and cover common architectures, but may not support your exotic layer or quantization scheme.
• Compilers automate fusion and code generation, but may miss peak performance on attention or communication.

For most teams, the right answer is to use a mature engine and spend engineering effort on workload characterization, scheduling and capacity. Kernel work is worth it when you have a differentiated model or a scale where a 10% gain pays for the team.

Next: quantization — how fewer bits change the memory equation, and where the quality and kernel support bills come due.