The model works in a notebook. In production, it has to survive a rolling deploy, a traffic spike, a full KV cache, a noisy tenant and a 3 a.m. page. That is a different discipline.

The serving control plane

A production inference stack has more moving parts than the engine:

• Gateway — authentication, rate limits, request validation, model routing.
• Load balancer — queue-aware, cache-aware distribution across replicas.
• Engine replicas — one or more GPUs per replica, with a model and tokenizer pinned.
• Autoscaler — adjusts replica count based on demand signals.
• Model registry — versioned weights, tokenizer, template, quantization config.
• Observability — metrics, traces, logs, cost attribution.
• Warm pool — preloaded capacity for fast scale-out.
• Fallback path — a smaller model or degraded mode when capacity is exhausted.

Each piece can fail independently. Design for partial failure, not just uptime.

Autoscaling is hard on GPUs

CPU autoscaling reacts in seconds. GPU autoscaling can take minutes:

1. A node is provisioned.
2. The container image is pulled.
3. The model weights are downloaded or mounted.
4. The engine starts and loads weights into HBM.
5. Kernels are compiled or loaded.
6. The replica becomes ready.

For a 70B model, this can be tens of minutes. If you scale only when the queue is already deep, you have already violated the SLO. Options:

• Warm pools — keep a small number of idle-ready replicas.
• Pre-pulled images and cached weights — remove download time.
• Fast model loading — memory-mapped weights, tensorized checkpoints, parallel loading.
• Predictive scaling — scale on time-of-day patterns, not just current load.
• Queue-aware scaling — scale on waiting requests and TTFT, not GPU utilization.
• Preemption / priority — protect latency-sensitive traffic while batch work waits.

A useful scaling signal combines queue depth, p95 TTFT and KV-cache occupancy. GPU utilization alone is too coarse: a saturated GPU might be serving one giant request while 100 users wait.

Load balancing with state

Round-robin load balancing assumes replicas are interchangeable. In LLM serving, they are not:

• Each replica has a different KV-cache state.
• Prefix-cache hits are local.
• A replica may be decoding a long request while another is idle.
• Tensor-parallel groups must receive requests as a unit.

A better router tracks:

• queue depth per replica,
• cache occupancy and prefix affinity,
• model and quantization version,
• health and readiness,
• tenant quotas.

Cache-aware routing is especially valuable for chat and agents with long shared system prompts. Sending a request to a replica that already has the prefix avoids a costly prefill. The router becomes a cache directory, not just a traffic cop.

Rolling updates

A model update is a new deployment, not a config change. The tokenizer, chat template and weights must move together. A mismatch can produce garbage output that looks like a model quality regression.

A safe rollout:

1. Pin the artifact — weights, tokenizer, template, engine version, quantization config.
2. Run shadow traffic — send a copy of requests to the new model, compare outputs and latency, but do not serve them.
3. Canary a small percentage — route real traffic, monitor quality and SLOs.
4. Warm the cache — if prefix caching matters, pre-populate common prefixes.
5. Shift traffic gradually — watch TTFT, error rate, quality and cost.
6. Keep the old replica warm — roll back quickly if needed.
7. Retire the old version — after the new one has survived a full traffic cycle.

Never compare two models on different prompts and call it an upgrade. Use the same eval set, the same sampling parameters and the same scoring.

Multi-tenancy

A shared model endpoint serves multiple customers. Without isolation:

• One tenant’s long prompts can evict another’s cache.
• One tenant’s retry storm can starve everyone.
• One tenant’s abuse can exhaust the GPU fleet.
• One tenant’s data can leak through shared cache blocks if the engine is not careful.

Controls:

• Per-tenant rate limits and concurrency caps.
• Per-tenant cache quotas.
• Priority classes and fair queueing.
• Request size limits — max prompt and output tokens.
• Cost budgets and alerts.
• Namespaced prefix caching — no cross-tenant block sharing.
• Audit logs for who called what, when, with which model.

Isolation is not just fairness; it is security.

Observability

Instrument the request path, not just the GPU:

• Gateway: request rate, auth failures, rate-limit rejections, payload sizes.
• Router: queue depth, routing decisions, cache affinity, retries.
• Engine: TTFT, TPOT, ITL, tokens/s, KV occupancy, preemptions, cache hit rate.
• Model: version, quantization, tokenizer, template.
• Cost: GPU-hours, tokens, cost per request and per tenant.
• Quality: eval scores, refusal rate, format errors, tool-call failures.

Use traces to connect an inference request to the product action that triggered it. An agent’s slow tool call may look like a model latency problem until you see the span.

Incident playbook

When inference is degraded:

1. Is it queueing, compute, memory or network? Check queue depth, TTFT, KV occupancy, bandwidth and error rates.
2. Is one tenant or model responsible? Segment by tenant, route and model version.
3. Is the cache full? Look for preemptions, evictions and a drop in cache hit rate.
4. Is a long prompt blocking the batch? Check prefill size and chunking.
5. Is a replica unhealthy? Check health, logs and GPU errors.
6. Is the router sending traffic to the wrong place? Check cache affinity and queue-aware routing.
7. Can you shed load safely? Reduce batch work, lower max output tokens, route to a smaller model, or reject with a retry hint.
8. Can you roll back? If the incident started with a deploy, roll back first and investigate second.

Write this down before you need it. Incident response is not the time to invent a dashboard.

Cost controls

Inference cost is driven by utilization, model size, precision, context length and output length. Levers:

• Right-size the model per route. Not every request needs the largest model.
• Cap output tokens. A runaway generation is a cost incident.
• Use prefix caching for repeated prompts.
• Batch offline work on spot or reserved capacity.
• Quantize where quality allows.
• Route simple steps to cheaper models with evals as a guardrail.
• Set budgets and alerts per tenant and per route.
• Track cost per completed task, not just cost per token.

Next: the inference capstone. You will size a stack from a workload brief, defend the tradeoffs, and write down what you would measure first.