Not marketing definitions. Each entry explains what the thing is and why it changes a decision. Search, filter, or jump from a lesson using the dotted terms in the prose.
The process of running a trained model forward to produce an output. In LLMs, inference is usually split into prefill and decode.
Why it matters: Training makes the weights; inference is where latency, throughput and cost become real.
The learned parameters of a neural network. Their number and precision determine how much memory the model needs.
Why it matters: Every decode step streams weights from memory, so weight size is a direct latency and cost input.
The individual numbers inside a model — attention projections, MLP matrices, embeddings and norms.
Why it matters: Parameter count sets the baseline compute and memory footprint, but active parameters matter more for MoE.
A chunk of text produced by a tokenizer. A token can be a word, part of a word, punctuation, or whitespace.
Why it matters: Models read and write tokens, not characters or words. Prices, context limits and latency are all token-based.
The deterministic mapping between text and integer token IDs. Different models may use different vocabularies.
Why it matters: Tokenization affects prompt length, cost and how models see code, numbers and non-English text.
The full set of tokens a model can recognize and emit, usually 32k–256k entries.
Why it matters: A larger vocabulary can shorten text but increases the output projection and can change model behavior.
The phase that processes the entire prompt in parallel and fills the KV cache before the first output token.
Why it matters: Prefill is compute-bound and sets time-to-first-token. Long prompts make it expensive.
The autoregressive phase that generates one token at a time, reusing the KV cache from prefill.
Why it matters: Decode is often memory-bandwidth-bound; its speed determines tokens per second and user-perceived fluency.
Unnormalized scores over the vocabulary produced by the model before softmax.
Why it matters: Sampling and constrained decoding operate on logits. Their shape is one reason generation is expensive.
Choosing the next token from a probability distribution instead of always taking the highest-scoring one.
Why it matters: Sampling controls creativity, determinism and safety. Greedy decoding is not the same as a reproducible system.
A scaling factor on logits that makes the distribution sharper (low) or flatter (high).
Why it matters: Low temperature is not zero randomness, and high temperature does not guarantee quality. It changes the shape of the output distribution.
Nucleus sampling: keep the smallest set of tokens whose cumulative probability reaches p, then sample from that set.
Why it matters: It removes the long tail without fixing the number of candidates. It is a common production default.
Stored key and value tensors from attention, so the model does not recompute the whole prefix for every new token.
Why it matters: The cache makes long-context generation possible, but it consumes memory that otherwise could serve more concurrent users.
The maximum number of tokens a model can attend to at once, including prompt, history and generated output.
Why it matters: It is a budget, not a target. Long contexts increase latency, memory and attention cost, and models do not attend equally everywhere.
Time to first token: the delay from request arrival to the first streamed output token.
Why it matters: TTFT is the latency users feel before anything appears. It includes queueing, tokenization and prefill.
Time per output token: the average interval between generated tokens after the first one.
Why it matters: TPOT controls how fast text streams and how long a response takes overall.
Inter-token latency: the time between consecutive tokens, including any scheduling gaps.
Why it matters: ITL captures jitter that average TPOT hides. A stream that pauses is worse than a consistently slower one.
The time a single request takes from arrival to completion.
Why it matters: Optimizing average latency can hide a terrible tail. Production promises are usually made in percentiles.
The number of tokens or requests completed per unit of time.
Why it matters: Throughput pays for the GPUs; latency wins the user. The two trade off through batching.
The portion of throughput that actually meets the latency SLO. Work that finishes too late does not count.
Why it matters: A system can report high tokens per second while failing most users. Goodput is the honest capacity metric.
A service-level objective: a measurable target such as “95% of requests under 800 ms TTFT”.
Why it matters: SLOs turn vague “fast” into a testable constraint for scheduling, capacity and cost.
Grouping multiple requests into one model step so shared weights and kernels are reused.
Why it matters: Batching raises throughput but can raise latency. The scheduler decides who waits.
A scheduling strategy that admits new requests as slots free up, instead of waiting for a fixed batch to finish.
Why it matters: It keeps GPUs busy under real traffic and is a core feature of modern serving engines.
Forming a fixed batch, running it to completion, then forming the next one.
Why it matters: Simple but wasteful: short requests wait for long ones, and new arrivals wait for the whole batch.
The component that decides which requests run, in what order, and when they enter prefill or decode.
Why it matters: Scheduling is where latency, fairness, memory and cost are traded every millisecond.
Reusing the KV cache for a shared prompt prefix across requests.
Why it matters: System prompts and few-shot examples are repeated constantly; caching them can cut prefill and cost dramatically.
A prefix-cache design that stores shared prompt prefixes in a radix tree and reuses them across requests.
Why it matters: It makes cache hits an explicit scheduling concern and is central to SGLang’s design.
Splitting a long prompt into chunks that can be interleaved with decode steps.
Why it matters: It prevents one long prompt from blocking all other users and smooths out latency.
Managing KV cache in fixed-size blocks with a block table, similar to virtual memory paging.
Why it matters: It reduces fragmentation and enables flexible sharing and preemption. It is the idea behind vLLM’s original scheduler.
A small draft model proposes several tokens, and the target model verifies them in parallel.
Why it matters: It can speed up memory-bound decode without changing the target model’s output distribution.
The fraction of drafted tokens accepted by the target model during speculative decoding.
Why it matters: The speedup depends on acceptance rate and draft cost; a plausible draft can still be a net loss.
Keeping the top-k partial sequences at each step instead of committing to one path.
Why it matters: It can improve some structured tasks but multiplies compute and memory; it is rarely the right default for chat.
Restricting the next-token distribution to satisfy a grammar, schema or allowed set.
Why it matters: It turns “please output JSON” into a guarantee at the decoder, at the cost of extra machinery and possible quality tradeoffs.
Representing weights or activations with fewer bits, such as FP8 or INT4, to save memory and bandwidth.
Why it matters: It can make decode faster and fit larger models, but quality and speed depend on the method and workload.
An 8-bit floating-point format with enough range for many inference workloads.
Why it matters: It halves weight and KV memory versus BF16 and is supported on newer GPUs, but needs scaling and kernel support.
A 4-bit integer format used for aggressive weight compression.
Why it matters: It can fit large models on small GPUs, but may hurt quality and does not automatically improve throughput.
Activation-aware weight quantization, which protects salient weights based on activation statistics.
Why it matters: It is a common 4-bit weight-only method for serving, balancing quality and memory.
A post-training quantization method that uses approximate second-order information to quantize weights.
Why it matters: It popularized 4-bit weight-only models and remains a baseline for quality/size tradeoffs.
Storing the KV cache in lower precision, such as FP8 or INT8, to serve more concurrent context.
Why it matters: KV memory often limits concurrency before weights do; quantizing it can be a large capacity win if quality holds.
Training a smaller model to imitate a larger model’s outputs or distributions.
Why it matters: It can produce a cheaper, faster model for a narrow task, but it is a training project, not a serving flag.
Removing weights, heads, layers or channels that contribute little to model output.
Why it matters: Unstructured pruning saves memory only with sparse kernels; structured pruning changes the architecture itself.
An architecture where each token is routed to a subset of expert MLPs instead of using all parameters.
Why it matters: MoE increases total capacity while keeping active compute per token lower, but introduces routing, load balance and all-to-all costs.
The mechanism that lets each token weigh information from other tokens using queries, keys and values.
Why it matters: Attention is where the KV cache, long-context cost and much of the memory traffic come from.
Sharing key/value heads among groups of query heads, reducing KV cache size versus full multi-head attention.
Why it matters: GQA is a major reason modern long-context models are servable at reasonable concurrency.
A single key/value head shared by all query heads.
Why it matters: It shrinks the KV cache drastically but can reduce quality; GQA is often the compromise.
An attention variant that compresses keys and values into a latent representation before caching.
Why it matters: It reduces KV cache memory for very large models, but serving support and implementation details matter.
An IO-aware attention kernel that tiles computation to avoid materializing the full attention matrix in high-bandwidth memory.
Why it matters: It makes attention faster and more memory-efficient, especially at long sequence lengths.
A captured sequence of GPU operations replayed with low CPU launch overhead.
Why it matters: Decode involves many tiny kernels; launch overhead can dominate unless the graph is captured.
Combining multiple GPU operations into one kernel to reduce memory traffic and launch overhead.
Why it matters: At decode time, the bottleneck is often data movement, not arithmetic; fusion reduces both.
A performance model that bounds throughput by either compute peak or memory bandwidth, depending on arithmetic intensity.
Why it matters: It explains why some workloads are compute-bound (prefill) and others are memory-bound (decode).
The ratio of floating-point operations to bytes moved for a kernel or workload.
Why it matters: It determines whether a GPU is limited by math or by memory bandwidth.
The rate at which data can be read from and written to GPU memory, usually in TB/s.
Why it matters: Decode streams weights and KV cache every token; bandwidth is often the real ceiling.
High-bandwidth memory stacked beside the GPU die, used for weights, activations and KV cache.
Why it matters: GPU capacity and bandwidth are HBM properties, and they set the serving envelope.
A GPU’s core execution block, containing CUDA cores, tensor cores, registers and shared memory.
Why it matters: Kernel efficiency is about keeping SMs fed, not just counting them.
A GPU unit specialized for matrix multiply-accumulate operations.
Why it matters: Tensor cores make prefill fast; decode often cannot feed them enough work because it is memory-bound.
A high-bandwidth GPU-to-GPU interconnect used for tensor and pipeline parallelism.
Why it matters: Multi-GPU serving is often communication-bound; NVLink changes what is practical.
The general-purpose bus connecting GPUs, CPUs and devices.
Why it matters: It is much slower than NVLink. Moving tensors across PCIe can erase the benefit of more GPUs.
Splitting individual weight matrices across GPUs and combining partial results, usually with all-reduce.
Why it matters: It fits models too large for one GPU and reduces per-GPU compute, but adds communication on every layer.
Assigning different layers of a model to different GPUs and passing activations between stages.
Why it matters: It reduces per-GPU memory without all-reducing every layer, but creates pipeline bubbles and latency.
Replicating the model across GPUs or nodes and distributing independent requests.
Why it matters: It scales throughput for models that fit on one device, but does not help a model that does not fit.
Distributing MoE experts across GPUs and routing tokens to the devices that hold the selected experts.
Why it matters: It is necessary for large MoE models and introduces all-to-all communication and load balancing challenges.
Running prefill and decode on separate GPU pools with KV cache transferred between them.
Why it matters: It lets each phase use hardware and parallelism that fit its bottleneck, at the cost of KV transfer and complexity.
Adding or removing serving capacity based on demand signals such as queue depth or tokens per second.
Why it matters: GPU cold starts are slow; scaling on CPU-style utilization often reacts too late.
The time to start a new model server, load weights and become ready to serve.
Why it matters: It can be minutes for large models, so autoscaling needs warm pools and load-balancing strategy.
The software that schedules requests, manages KV cache, runs kernels and exposes an API around a model.
Why it matters: vLLM, SGLang, TensorRT-LLM and others make different tradeoffs; the engine is a core architectural choice.
Requests waiting for compute, memory or a scheduling slot.
Why it matters: Queue time is part of TTFT and grows nonlinearly as utilization approaches 100%.
Model FLOPs utilization: the fraction of a GPU’s peak FLOPs actually achieved by a model.
Why it matters: It turns headline hardware specs into realistic prefill estimates. 30–50% is often more honest than 100%.
Model bandwidth utilization: the fraction of peak memory bandwidth achieved during a workload.
Why it matters: Decode estimates are only as good as the bandwidth efficiency you assume.
The runtime layer around a model that supplies tools, context, permissions, state, retries and stopping conditions.
Why it matters: The same model can be a chatbot or an autonomous agent depending on its harness.
A system that observes an environment, decides on actions, uses tools and iterates toward a goal.
Why it matters: Agency is a property of the loop and permissions, not a special model checkpoint.
The repeated cycle of observing state, choosing an action, executing a tool and incorporating the result.
Why it matters: Most agent failures are loop failures: no progress, no verification, or no stop condition.
A prompting and control pattern that interleaves reasoning traces with tool actions and observations.
Why it matters: It is a useful baseline, but reasoning text is not a guarantee of correct execution.
An agent pattern that creates a plan first, then executes steps, optionally replanning when state changes.
Why it matters: It improves long-task structure but can become brittle if the plan is never revised.
A structured request from the model to run an external function with typed arguments.
Why it matters: Tools are the agent’s hands. Their schemas, errors and permissions shape what it can do safely.
The name, description, parameters and constraints that tell a model how to call a tool.
Why it matters: A vague schema produces vague calls. Error messages are part of the interface, too.
Model Context Protocol: an open protocol for exposing tools, resources and prompts to AI applications.
Why it matters: It standardizes integrations, but does not remove the need for permissions, sandboxing and tool review.
Designing what enters the model’s context, in what order, and what gets removed or summarized.
Why it matters: Context is the agent’s working memory. More text is not automatically better; relevance and structure matter.
Reducing long conversation or tool history into a shorter summary while preserving the information needed to continue.
Why it matters: It buys context headroom but can silently delete constraints, so compaction policy is a correctness decision.
Rewriting a longer history into a shorter representation, usually with a model or deterministic rules.
Why it matters: Summaries can be lossy. Pinned facts, decisions and unresolved errors should survive separately.
Fetching relevant external information into context instead of keeping everything in the prompt.
Why it matters: It scales memory beyond the context window but introduces recall, freshness and ranking failures.
A separate agent context invoked by a parent agent for a scoped task, returning a result.
Why it matters: It isolates context and parallelizes work, but adds handoff loss, cost and coordination overhead.
Passing control and a compressed state summary from one agent or session to another.
Why it matters: Most multi-agent failures happen at the handoff: missing constraints, stale state or ambiguous ownership.
A persistent conversation or task context with an identity, state and history.
Why it matters: Sessions turn one-shot calls into work that can be inspected, resumed and audited.
The durable facts an agent needs: files, tool results, decisions, pending approvals and progress.
Why it matters: If state lives only in the model’s context, a compaction or crash can erase the task.
A saved snapshot of task state that can be resumed or inspected later.
Why it matters: Checkpoints make long-running agents recoverable and make failures reproducible.
An append-only record of actions, observations and decisions in a session.
Why it matters: It is the source of truth for replay, audit, evals and debugging — not just a transcript.
A policy that decides which tool actions run automatically, require approval, or are denied.
Why it matters: Autonomy should be granted per action and per scope, with a clear escalation path.
An isolated execution environment that limits filesystem, network, process and secret access.
Why it matters: A model instruction is not a security boundary. The sandbox is.
A human decision to allow a specific action or scope of actions.
Why it matters: Approvals reduce risk but add latency and fatigue; good design makes them rare, specific and informative.
A workflow where a person reviews or authorizes steps the agent cannot safely take alone.
Why it matters: It is a control, not a decoration. The human needs enough context to make a real decision.
Malicious or unintended instructions embedded in content the model reads, attempting to redirect its behavior.
Why it matters: Any tool output can carry instructions. Treat external text as data, not authority.
Moving sensitive data out of an allowed boundary, intentionally or through a manipulated agent.
Why it matters: An agent with network and secret access can leak data even without a code vulnerability.
Malicious tool descriptions, metadata or outputs that influence the model to misuse a tool.
Why it matters: Tool registries are supply-chain surface. Review and pin what the agent can call.
Giving an agent the minimum access needed for the task, for the shortest time.
Why it matters: Broad credentials turn a small model mistake into a large incident.
An explicit set of permitted commands, domains or tools; everything else is denied.
Why it matters: Allowlists are easier to reason about than blocklists, especially for shell and network access.
A structured record of a request or task, usually composed of spans for model calls, tools, retries and approvals.
Why it matters: Without traces you can only see the final answer, not why the harness succeeded or failed.
A timed operation inside a trace, such as one model call or one tool execution.
Why it matters: Spans expose where latency, cost and errors actually happen.
The ability to understand a system’s internal behavior from its outputs: logs, metrics, traces and events.
Why it matters: Agent incidents are rarely reproducible from a single error message.
Systematic tests that measure whether a model or harness meets a defined standard on a dataset of tasks.
Why it matters: Demos show possibility; evals show whether a change made the system better.
Evaluating the sequence of actions and observations an agent took, not only its final answer.
Why it matters: A correct answer from a dangerous or wasteful path is a latent failure.
Evaluating whether the final state or answer satisfies the task, independent of the path taken.
Why it matters: Outcome and trajectory evals answer different questions and should usually be paired.
Using a model to score outputs against a rubric or reference.
Why it matters: It scales subjective evaluation but introduces bias, drift and prompt sensitivity. Calibrate it against humans.
A fixed set of tasks run after every change to catch previously solved failures from returning.
Why it matters: Harness changes are software changes; they need tests with stable expectations.
Sending each request or step to a model chosen for its cost, latency or capability.
Why it matters: Most agent steps do not need the strongest model; routing can dominate cost savings if evals keep quality honest.
A cap on input, output or total tokens for a task or step.
Why it matters: Budgets force prioritization and prevent a looping agent from spending indefinitely.
The maximum number of agent iterations or tool calls allowed before the harness stops.
Why it matters: It is a cheap, essential guardrail against loops and runaway cost.
Running a failed operation again, ideally with feedback or a changed strategy.
Why it matters: Retries improve reliability only when the operation is safe to repeat and the failure is transient.
Waiting progressively longer between retries to avoid overwhelming a failing service.
Why it matters: Immediate repeated retries turn a transient error into a retry storm.
A property where repeating an operation has the same effect as running it once.
Why it matters: Retries and resumptions are only safe when tool calls are idempotent or keyed.
A maximum time allowed for an operation before it is cancelled or treated as failed.
Why it matters: Without timeouts, a stuck tool can occupy a session indefinitely.
Stopping an in-flight operation when it is no longer needed or has exceeded its budget.
Why it matters: Agents need a real stop button that propagates to tools, not just a UI spinner.
Detecting repeated actions or states that indicate the agent is stuck.
Why it matters: A loop can burn tokens and time while looking productive. Compare state, not just messages.
Re-running a recorded trace with the same inputs and tool results to reproduce a failure.
Why it matters: Replay turns a one-off incident into a debuggable test case.
Producing the same output for the same inputs and state.
Why it matters: LLM sampling is not deterministic by default, so harnesses need seeds, pinned versions and recorded tool outputs where possible.
A constraint outside the model that blocks, modifies or flags dangerous behavior.
Why it matters: Guardrails must be enforceable in code and policy, not just requested in a prompt.
The harness component that decides what the agent may do, where, and under whose authority.
Why it matters: It separates capability from permission, which is the core safety architecture of an agent.
A packaged instruction and resource set that an agent can load for a specific kind of task.
Why it matters: Skills are context engineering as a product feature: reusable expertise without permanent prompt bloat.
A deterministic callback that runs before or after a tool action, model call or session event.
Why it matters: Hooks let a harness enforce policy, add telemetry or transform output without relying on the model.
A packaged extension that adds tools, commands, providers or UI behavior to a harness.
Why it matters: Plugins expand the attack surface and the maintenance surface; version and permission them.
The degree to which an agent can act without human approval.
Why it matters: Autonomy should be scoped per action and environment, not treated as a single on/off preference.
Evidence that the task was completed correctly, such as tests, a diff review or a deterministic check.
Why it matters: An agent saying “done” is not verification. The harness needs an independent signal.
The concrete conditions that must be true before an agent may stop: tests pass, docs updated, diff reviewed.
Why it matters: Stopping conditions shape behavior more than most prompt instructions.