You do not need a PhD to understand why a model is slow, why an agent loops, or why the bill doubled. You need a clear picture of what happens between a prompt and a token — and between a token and an action.

This site teaches two disciplines that are usually learned separately:

• Inference engineering — turning trained weights into a reliable, affordable stream of tokens.
• Harness engineering — turning a stream of tokens into work that can be trusted.

They meet at the same boundary. Inference decides how fast and how cheaply a model can think. The harness decides what the model is allowed to do with that thought, what it remembers, and when the work is actually finished.

The 60-second version

A language model is a very large function. You give it a sequence of integers and it returns a probability distribution over the next integer. Then you pick one, append it, and repeat.

Those integers are tokens. A tokenizer chops text into pieces — sometimes a whole word, sometimes a fragment, sometimes a single space or punctuation mark. The model never sees characters. It sees token IDs.

The model itself is a stack of transformer blocks. Each block has an attention layer and an MLP. Attention lets each position look at earlier positions. The MLP transforms each position independently. Residual connections and normalization hold the stack together. At the end, a language-model head projects the final hidden state onto the vocabulary.

There are two phases:

1. Prefill reads the prompt. All prompt tokens are processed in parallel. This phase is usually compute-bound, and it fills the KV cache — the stored keys and values that later tokens will attend to.
2. Decode produces output one token at a time. Each step reads the weights and the growing KV cache, does relatively little math per byte moved, and appends one token. This phase is usually memory-bandwidth-bound.

That distinction explains most serving behavior. Prefill sets time to first token (TTFT). Decode sets time per output token (TPOT) and therefore how fast text streams. When people say “the model is slow,” one of those two numbers is usually bad — and they have different fixes.

Inference is a resource-allocation problem

A serving system has four budgets, and they push against each other:

You can almost always improve one by making another worse. A bigger batch raises throughput and lowers cost per token, but it can raise TPOT and hurt tail latency. A longer context improves capability, but the KV cache eats the memory that would have served more users. A cheaper model lowers token cost, but may need more retries.

This is why the inference track keeps returning to the same question: what is the workload, and what does “good” mean for it? A benchmark without an SLO is a science experiment, not a capacity plan.

A harness is the rest of the product

A raw model call takes text and returns text. An agent takes a goal, observes an environment, calls tools, remembers what happened, and decides when to stop. The layer that makes this possible is the harness.

A harness has six recurring jobs:

• Loop — observe, plan, act, verify, repeat or stop.
• Tools — typed functions the model can call: read a file, run a test, search the web.
• Context — what the model sees on this turn, and what gets summarized or dropped.
• Policy — what the model may do, where, and with whose approval.
• State — durable facts, checkpoints, and an event log that survives a crash or a compaction.
• Observability — traces, costs, evals, and replay.

Most agent failures are not “the model was too small.” They are missing context, missing verification, or missing authority boundaries. A harness that gives a model shell access but no sandbox has not built an agent; it has built an incident generator.

The two tracks, side by side

The tracks are not independent. An agent’s tool calls are inference requests. A serving SLO is part of the product experience. A harness that retries five times because the model is cheap can be more expensive than one strong call. And an inference stack that streams beautifully is useless if the harness loops forever.

How to read this site

You can read linearly, or take a path:

• New to all of this: start here, then the inference track, then the harness track. Do the tokenizer lab early — it makes token counts concrete.
• You run models: start with The Life of a Token, then GPU anatomy, KV cache and batching. Use the Serving Workbench after each lesson.
• You build agents: start with What Is a Harness?, then the agent loop, context and permissions. Use the Harness Workbench to break things on purpose.
• You need a reference: the Stack Map shows how the layers connect. The Glossary has working definitions, not marketing ones.

Two themes are available in the header. Draft is a warm paper notebook: serif display type, ink lines, cobalt and orange. Live is a night signals desk: indigo surfaces, electric cyan, hot magenta and violet traces. They are the same information in two visual languages. Use whichever helps you think.

The one habit that matters

When something is slow, expensive or unreliable, do not start with a technique. Start with a boundary:

• Where does the request wait?
• What is in memory at that moment?
• Which budget is exhausted?
• What evidence would prove the system is working?

That habit is the whole site. The rest is detail — important, learnable detail, but detail.

In the next lesson, follow a single request from arrival to the last streamed token. That journey is the spine of the entire inference track.