A serving engine is a control plane for the GPU: it schedules requests, manages KV memory, dispatches kernels and exposes an API. The differences between engines are mostly differences in how they make those tradeoffs.

The common architecture

Almost every engine has:

• An API layer — often OpenAI-compatible, with auth, rate limits and chat templating.
• A scheduler — continuous batching, priority, admission control.
• A KV-cache manager — paged allocation, prefix caching, preemption.
• A kernel layer — attention, GEMM, sampling, quantization.
• A streaming layer — tokenization, detokenization, backpressure.
• Observability — metrics, traces, logs.

If you understand these pieces, an unfamiliar engine is a set of answers to familiar questions.

The engines

vLLM

vLLM popularized PagedAttention and continuous batching. It is widely used, open source, and supports a broad range of models and hardware. Its strengths:

• Strong default scheduling and memory management.
• Large model ecosystem and OpenAI-compatible server.
• Prefix caching, chunked prefill, speculative decoding, quantization.
• Active community and integrations (Kubernetes, llm-d, KServe).

Tradeoffs:

• Kernel performance varies by model and quantization.
• Some advanced features require configuration and tuning.
• Multi-node and MoE support depend on version and hardware.

SGLang

SGLang grew out of structured generation and prefix sharing. Its RadixAttention design makes prefix caching a first-class scheduler concern. It is strong for:

• Workloads with shared system prompts, tool definitions or few-shot examples.
• Structured outputs and complex generation programs.
• High-throughput serving with prefix reuse.

Tradeoffs:

• The programming model and ecosystem differ from vLLM.
• Kernel coverage and model support vary.
• Operational maturity depends on your deployment target.

TensorRT-LLM

NVIDIA’s stack compiles models into optimized engines. It can deliver excellent performance on NVIDIA hardware, especially with FP8/INT4 and in-flight batching. It is strong for:

• Maximum performance on supported NVIDIA GPUs.
• Teams already invested in NVIDIA tooling.
• Production deployments that can afford a build step.

Tradeoffs:

• Engine compilation adds complexity and build time.
• Model and feature support lags the open ecosystem.
• Less portable across vendors.

Text Generation Inference (TGI)

Hugging Face’s serving stack. It is well integrated with the Hugging Face model ecosystem and offers a straightforward path from model to endpoint. Strengths include broad model support, quantization options and production features. Tradeoffs include performance variability by model and a smaller advanced-scheduling surface than vLLM or SGLang.

llama.cpp

A C/C++ implementation optimized for local and edge inference. It supports GGUF quantization, CPU and GPU backends, and runs on laptops, workstations and small servers. It is excellent for:

• Local development and offline use.
• Heterogeneous hardware.
• Aggressive quantization on limited memory.

It is not designed for high-concurrency data-center serving. Its scheduling and batching are simpler than vLLM’s or TensorRT-LLM’s.

Others

LMDeploy, MLC-LLM, OpenVINO, and vendor-specific stacks each have niches. The evaluation questions remain the same.

The axes that matter

No engine wins on every axis. A team optimizing a single model on a fixed GPU fleet may prefer TensorRT-LLM. A team serving many open models may prefer vLLM. A team with heavy prefix reuse may prefer SGLang. A team running on laptops may prefer llama.cpp.

How to benchmark honestly

A generic “tokens per second” chart is almost useless. Benchmark your workload:

1. Fix the model and precision. Compare the same checkpoint or a documented equivalent.
2. Use realistic prompt and output length distributions. A uniform 128-token benchmark hides long-context behavior.
3. Measure TTFT, TPOT, ITL and goodput, not just aggregate throughput.
4. Sweep concurrency. Performance at batch 1 and batch 256 tells different stories.
5. Include prefix-cache behavior. If your prompts share prefixes, test with and without caching.
6. Include the full path. Tokenization, scheduling, streaming and network.
7. Run long enough to see tail latency. Short bursts miss queueing and memory pressure.
8. Record the exact versions. Engines change quickly; a benchmark without a commit hash is folklore.

Deployment shape

A serving deployment is more than an engine process:

• Gateway / router — auth, rate limits, model routing, canonical API.
• Engine replicas — one process per GPU or GPU group.
• Load balancer — queue-aware, cache-aware routing.
• Autoscaler — scales on queue depth, TTFT and cache pressure, not just GPU utilization.
• Model registry — versioned weights, tokenizer, chat template, quantization config.
• Observability — metrics, traces, logs, cost attribution.
• Warm pool — preloaded models to avoid cold-start latency.

Kubernetes is common, but it is not required. A small deployment can run on a few VMs with a reverse proxy. The important thing is that the routing layer understands the engine’s state. Sending requests blindly to a replica with a full KV cache is a recipe for tail latency.

A selection procedure

1. Write down the workload: model, context distribution, output distribution, concurrency, SLO, hardware, budget.
2. Shortlist two or three engines that support the model and hardware.
3. Build a representative benchmark with your prompts and a load generator.
4. Measure goodput and tail latency at several concurrency levels.
5. Test failure modes: model load, cache exhaustion, a long prompt, a rolling restart.
6. Check operations: metrics, logs, upgrade path, community.
7. Run a pilot with real traffic before committing.

The engine is a long-lived dependency. Choosing one on a single benchmark number is like choosing a database on insert speed alone.

Next: benchmarking and capacity — how to turn measurements into a defensible number of GPUs.