A 70B model does not fit on an 80 GB GPU in BF16. A 671B MoE model does not fit on eight of them without a plan. Parallelism is the plan — and the plan is mostly about communication.

Four axes

Most large deployments use a combination: DP for throughput, TP within a node for memory and compute, PP across nodes when needed, EP for sparse models.

Tensor parallelism

Tensor parallelism splits a weight matrix across GPUs. For a column-parallel linear layer, each GPU computes part of the output columns. For a row-parallel layer, each GPU computes a partial sum that must be reduced. Attention heads can be split naturally; MLP intermediate dimensions can be split too.

The catch: every layer needs a synchronization. In the common Megatron-style pattern, there is an all-reduce after the attention output projection and another after the MLP down projection. At 32 layers, that is dozens of collective operations per token.

All-reduce cost depends on message size and interconnect. NVLink 4 on an H100 offers about 900 GB/s bidirectional per GPU in a DGX; PCIe Gen5 is roughly an order of magnitude slower. Tensor parallelism across NVLink is practical. Tensor parallelism across PCIe or Ethernet is often communication-bound.

The scaling is sublinear even on NVLink. If one GPU produces 100 tokens/s, two GPUs do not produce 200. They produce something less, because of communication and reduced per-GPU work. The exact curve depends on model size, batch size and interconnect. Measure it.

Pipeline parallelism

Pipeline parallelism assigns contiguous layer ranges to different GPUs. GPU 0 runs layers 1–10, GPU 1 runs 11–20, and so on. Activations flow between stages.

The advantage: no all-reduce on every layer; only stage boundaries need communication. The disadvantage: pipeline bubbles. While GPU 0 processes a micro-batch, GPUs 1–3 wait. To fill the pipeline, you need multiple micro-batches in flight, which increases memory and latency.

Pipeline parallelism is often used across nodes, where NVLink is unavailable. It is less attractive within a node where TP is possible. It also complicates scheduling: a request occupies every stage, and the slowest stage sets the throughput.

Data parallelism

Data parallelism replicates the model on multiple GPUs or nodes and sends different requests to each replica. It is the simplest way to scale throughput for a model that fits on one device.

For serving, DP has two important nuances:

• Routing matters. A load balancer that ignores prefix-cache locality can send a request to a replica that must recompute a prefix another replica already has.
• Tail latency matters. A request routed to a busy replica waits, even if another replica is idle. Work-stealing or queue-aware routing helps.

DP does not help a model that does not fit. It scales capacity, not model size.

Expert parallelism and MoE

A mixture-of-experts model replaces each dense MLP with a set of expert MLPs and a router. For each token, the router selects the top-$k$ experts (often 2 of 8, or 8 of 256). Only those experts compute.

This changes the economics:

• Total parameters can be enormous, because capacity is stored across many experts.
• Active parameters per token are much smaller, so compute per token can be manageable.
• Memory still scales with total parameters, not active parameters. All experts must be resident somewhere.
• Routing introduces load imbalance. If one expert is hot, its GPU becomes the bottleneck.

Expert parallelism distributes experts across GPUs. Tokens are routed to the devices that hold their selected experts, usually via all-to-all communication. This is a different traffic pattern from TP’s all-reduce: it is bursty, data-dependent and sensitive to imbalance.

DeepSeek-V3 is a useful case study. It has 671B total parameters and about 37B active per token, with MLA attention and a large fine-grained expert set. Serving it efficiently requires expert parallelism, careful load balancing and communication overlap. The architecture is designed for this, but it is not a generic “just add GPUs” solution.

Communication as a first-class cost

A useful mental model:

If communication is the largest term, more compute does not help. This happens in several common situations:

• TP with small per-GPU work at low batch.
• PP with too few micro-batches, leaving bubbles.
• EP with imbalanced experts.
• Multi-node serving over slow interconnects.
• Disaggregated prefill/decode with KV-cache transfer.

Overlap is the main mitigation: communicate while computing. Tensor-parallel all-reduce can overlap with the next layer’s compute if the schedule allows. Expert all-to-all can overlap with local expert computation. Pipeline stages can overlap micro-batches. Good engines do this; naive implementations do not.

Disaggregated prefill and decode

Prefill and decode have different bottlenecks. Prefill is compute-heavy; decode is memory-heavy. Running them on the same GPUs forces a compromise. Disaggregated serving puts prefill on one pool and decode on another, transferring the KV cache between them.

Benefits:

• Each pool can use hardware and parallelism matched to its phase.
• Long prefills do not block decode streams.
• Scaling can be independent: add prefill capacity for long prompts, decode capacity for concurrency.

Costs:

• KV-cache transfer over the network. A 32K-token cache can be gigabytes; moving it is not free.
• More complex routing and scheduling.
• Two pools to operate, monitor and debug.
• Cache locality becomes harder.

Disaggregation is not automatically better. It wins when the workload has a clear phase imbalance or when prefill interference is a major latency problem. It loses when KV transfer dominates or when the workload is small and uniform.

A sizing checklist

When a model does not fit or does not meet its SLO on one GPU:

1. Can quantization make it fit? Try FP8 first, then weight-only INT4 if needed.
2. Can tensor parallelism fit within a node? Prefer NVLink-connected GPUs.
3. Do you need pipeline parallelism across nodes? Budget for bubbles and activation transfer.
4. Is the workload MoE? Plan for expert parallelism, all-to-all and load balancing.
5. Is prefill interfering with decode? Consider chunked prefill before full disaggregation.
6. Is the network the bottleneck? Measure collective time, not just GPU utilization.
7. Does the scheduler understand locality? Prefix-cache-aware routing can matter more than raw GPU count.

Next: serving engines — how vLLM, SGLang, TensorRT-LLM, TGI and llama.cpp differ, and how to choose without trusting a generic benchmark.