Stand on the sources.
The lessons synthesize primary documentation, papers and production experience. Where a number is an estimate, the lesson says so. Where a topic moves quickly, the source is linked so you can check the current state.
Hardware and performance models
The physical facts behind every estimate in the inference track.
Memory capacity, bandwidth and BF16 tensor throughput for the H100 family.
HBM3e capacity and bandwidth for the H200.
Blackwell architecture, NVLink 5 and B200 platform details.
The original roofline paper. Arithmetic intensity and the compute/memory ceiling.
Standardized inference benchmarks. Useful for comparing systems, with caveats about workload shape.
Transformers, attention and the KV cache
Architecture details that change serving behavior.
The transformer paper. Read it for the attention and residual structure, not serving advice.
Multi-query attention and the KV-cache bandwidth problem.
Grouped-query attention, the default KV-memory compromise in many open models.
Multi-head latent attention and compressed KV cache.
A large sparse MoE, MLA and the infrastructure behind it.
Serving engines and scheduling
How production systems turn a model into a service.
The vLLM paper: paged KV cache, memory waste and continuous batching.
Scheduler, prefix caching, quantization and OpenAI-compatible serving.
RadixAttention, prefix sharing and high-throughput serving.
RadixAttention and the case for prefix-aware scheduling.
NVIDIA’s optimized inference stack, kernels, quantization and in-flight batching.
Hugging Face’s serving stack and deployment patterns.
Local and edge inference, quantization formats and a very different set of tradeoffs.
Distributed inference on Kubernetes, including disaggregated prefill/decode patterns.
Optimization and decoding
The techniques that make tokens cheaper or faster — and their limits.
Tiling and IO awareness. Why attention speed is about memory movement.
The original speculative decoding formulation and acceptance-rate math.
A stronger draft strategy for speculative decoding.
Multiple decoding heads as an alternative to a separate draft model.
One of the foundational 4-bit weight-only quantization methods.
Protecting salient weights. A common serving choice for 4-bit models.
Balancing activation and weight quantization difficulty.
Harnesses, agents and tools
The loop, context and tooling layers around the model.
Practical patterns and the case for simple, composable workflows.
The observe-think-act pattern that most agent loops resemble.
Self-reflection as a learning signal, and its limits.
Early evidence that tool use can be learned rather than hand-wired.
The protocol behind many tool integrations. Read the security sections too.
Permission modes, subagents, hooks, skills and MCP integration.
Codex surfaces, sandboxing and approval modes.
Open-source coding harness: agents, tools, permissions and providers.
A focused coding agent with repository maps and edit formats.
IDE-native agent with tool approvals and MCP support.
An open platform for software agents and sandboxed execution.
Safety, traces and evaluation
How to make agent behavior observable, testable and bounded.
Private data + untrusted content + external communication. A useful threat model.
A structured catalog of application-layer risks, including prompt injection and excessive agency.
A vendor-neutral starting point for model and tool spans.
Eval design basics, with useful caveats about judges and datasets.
Books and longer courses
Where to go after the lessons.
The GPU programming foundation behind kernel-level intuition.
Production ML systems, data and operations. Not LLM-specific, but durable.
A rigorous course on building language models, useful for deeper architecture work.