Two tracks. One system.
Read linearly or jump to what you need. Inference starts with a single request and ends with a capacity plan. Harness starts with a model call and ends with a release engineer you would actually trust.
Start here
The mental model in one sitting. If you are new to the site, read this first.
Inference engineering
From a request to a serving plan. Tokens, GPUs, memory, schedulers, engines, capacity and production.
The Life of a Token
Follow one request from arrival to streamed output, and learn the four latency numbers that decide whether the experience feels fast.
The GPU Is a Memory Machine
SMs, tensor cores, HBM and the roofline — the physical facts that explain why decode is bandwidth-bound and prefill is compute-bound.
Attention, MLPs and Two Shapes
A serving-level tour of the transformer: what each block does, why attention needs a cache, and how the same weights produce two completely different performance regimes.
The Cache Is the Model
Why the KV cache grows the way it does, how to estimate it, and why memory — not compute — is usually what limits concurrency.
Batching Is a Scheduling Decision
Static batching wastes time. Continuous batching fills it. But every batch decision trades throughput against the latency someone feels.
Schedulers, Pages and Prefixes
Inside the memory manager: paged KV allocation, prefix reuse, chunked prefill, preemption, and the policies that keep latency from collapsing under load.
Kernels, Fusion and Graphs
The GPU does not run a model. It runs hundreds of kernels. How those kernels are fused, scheduled and launched decides whether the hardware is actually busy.
Quantization Without Fantasy
What actually gets smaller, what gets faster, and why “4-bit” is a family of methods rather than a single guarantee.
Sampling and Speculation
How a probability distribution becomes a token, and how speculative decoding buys several tokens per target-model pass without changing the target distribution.
Parallelism, MoE and the Network
When a model outgrows one GPU, the interconnect becomes part of the performance model. Tensor, pipeline, data and expert parallelism each buy something different.
Choosing a Serving Engine
vLLM, SGLang, TensorRT-LLM, TGI and llama.cpp solve the same problem with different priorities. Learn the axes that matter before you read a benchmark.
Benchmarking and Capacity
How to measure a serving stack without fooling yourself, and how to turn latency, throughput and goodput into a GPU count you can defend.
Production Inference
Autoscaling, cold starts, routing, rollouts, multi-tenancy and the incidents that only appear after launch.
Capstone: Size a Serving Stack
A complete capacity exercise — from a product brief to a GPU count, a cost estimate, a benchmark plan and a list of reasons the plan might be wrong.
Harness engineering
From a completion to verified work. The loop, tools, context, permissions, state, reliability, evals and cost.
What Is a Harness?
The model generates tokens. The harness turns those tokens into actions, memory and a stopping condition. Learn the six systems that make an agent.
The Agent Loop
Observe, plan, act, verify, repeat. The loop is simple; the hard part is making each turn change state and knowing when to stop.
Context and Compaction
The context window is the agent’s working memory. What you put in, what you leave out, and what you summarize decides whether it can finish the task.
Tools, MCP and Error Design
A tool is a contract, not a function. Its schema, error messages and side effects determine whether the agent can use it reliably.
Permissions and Sandboxing
A prompt is not a security boundary. Learn how to give an agent exactly the authority a task needs — and how to enforce it in code, not in a sentence.
State and Subagents
Long tasks need memory that outlives the context window. Learn the three kinds of state, how to checkpoint them, and when a subagent is worth the coordination cost.
Reliability Under Failure
Retries, timeouts, cancellation, idempotency and loop detection. How to make an agent survive slow tools, flaky networks and its own bad ideas.
Evals and Traces
Outcome evals tell you whether the task worked. Trajectory evals tell you how. Learn to instrument both without drowning in logs or fooling yourself with a judge.
Cost and Model Routing
The cheapest token is not the cheapest task. Learn to model cost per completed task, route work by difficulty, and keep a cheap model from becoming an expensive retry loop.
Harness Teardowns
Claude Code, Codex, OpenCode, Aider, Cline and OpenHands solve the same six problems differently. Read their choices as design arguments, not feature lists.
Build Your Own Harness
A minimal coding harness in five modules — loop, model adapter, tools, policy and trace. Small enough to understand, complete enough to run a real task.
Capstone: Design a Release Engineer
A complete harness design review — tools, permissions, state, evals, cost and failure modes for an agent that touches real repositories and real deploys.