The stack, from HBM to the human.
A request is not one system. It is a stack of constraints. Read it bottom-up: the hardware sets the physical budget, the model spends it, serving schedules it, the harness gives it authority, and the application makes a promise to a user.
The map is not a deployment diagram. It is a dependency diagram. A change at one layer shows up as pressure at another: a longer context becomes a KV-cache bill, a bigger batch becomes a latency tail, a new tool becomes an authority boundary.
Use it in both directions. When something is slow or expensive, walk down until you find the exhausted budget. When you add a capability, walk up and ask which layer has to pay for it.
Application
The product surface: chat, IDE, API, CLI, or an automated workflow.
Harness
The runtime that gives a model memory, tools, permissions, and a stopping condition.
Serving
The scheduler and runtime that turn model weights into tokens under a latency budget.
Model
The weights and the computation: attention, MLPs, experts, activations.
Hardware
The physical budget: SMs, HBM, bandwidth, NVLink, power, and money.
Memory is the first constraint.
Weights and KV cache share HBM. A 70B model in BF16 does not fit on one 80 GB GPU. At 32K context, the cache can be larger than the weights. Every capacity plan starts here.
KV cache lesson →Throughput is somebody's latency.
Batching amortizes weights but makes requests wait. Continuous batching fills the gaps. Chunked prefill protects streams. Every scheduling policy is a statement about whose latency matters.
Scheduling lesson →The prompt is not a boundary.
Tools turn tokens into action. Permissions, sandboxes and approvals decide which actions are allowed. If the model can talk the harness into anything, the model has the authority.
Permissions lesson →The interesting failures are between layers.
A fast model with a looping harness is still slow. A safe harness with a starved GPU is still unavailable. Learn both sides.