Multi-GPU Is a Communication Problem
Why splitting an LLM across GPUs is about the wires between them, not the FLOPs inside them
The FLOPs Were Never the Hard Part
A model that doesn't fit on one GPU forces you to split it. The matrix multiplies stay embarrassingly parallel — you can always buy more compute. What you cannot buy your way out of is the communication the instant the model spans devices.
Every tensor-parallel layer ends in an all-reduce. Every pipeline stage hands activations to the next over a link. Every Mixture-of-Experts layer shuffles tokens to their experts and back. The GPUs spend a startling fraction of wall-clock time waiting on each other. vLLM's multi-GPU performance work is, overwhelmingly, communication engineering.
The Interconnect Hierarchy
Four tiers matter, and they span five orders of magnitude in bandwidth. The right parallelism strategy is a function of which tier your GPUs talk over.
Tensor → NVLink
Hammers the fabric every layer. Belongs on the fastest tier.
Pipeline → PCIe / network
Communicates once per stage boundary. Tolerates slow links.
Expert → sparse all-to-all
Trades a dense all-reduce for a token-sparse shuffle.
Disagg → network, once
Moves the KV cache over the network exactly once per request.
The V1 Process Model
vLLM V1 (re-architected core, January 2025) runs multiple GPUs as a small constellation of OS processes. The parallelism flags are really instructions for how many of each process to spawn.
Total GPU workers = TP × PP × DP. A TP=2, PP=2 launch is 4 workers, one per GPU.
It runs the scheduler and KV-cache manager, and owns the TP × PP workers beneath it.
It keeps data-parallel ranks in lockstep. Turns out to be essential for MoE models (Module 04).
vLLM defaults to native Python multiprocessing single-node and Ray multi-node, overridable with --distributed-executor-backend {mp,ray}. Workers are stateful and symmetric — the scheduler ships only the per-step diff, keeping cross-process chatter small.
Check Your Understanding
What is the dominant cost that appears the moment a model is split across GPUs?
How many GPU worker processes does V1 launch?
Tensor Parallelism & the All-Reduce
Shard every layer, pay a collective twice per layer — then make it faster and make it overlap
Megatron Sharding, and the Bill
Tensor parallelism (--tensor-parallel-size) splits each weight matrix across GPUs. Every device computes a partial result; an all-reduce sums the partials back to a coherent activation. It's the default for splitting a model within a node — TP=4 on a 4-GPU box.
One after attention, one after the MLP — for every transformer layer, on every forward pass. On NVLink this is tolerable. On anything slower it dominates the step. So the whole TP optimization story is two moves: make the all-reduce faster, and make it overlap compute so it's partly free.
The Custom All-Reduce Fast Path — and Its Gates
vLLM ships a hand-written NVLink-optimized all-reduce that beats NCCL for the small-to-medium tensors typical of inference. But it is heavily gated. Reading the gates (from custom_all_reduce.py) tells you exactly when you get the fast path — and when you silently fall back to NCCL.
use_custom_ar = (
world_size in {2,4,6,8}
and (world_size == 2 or fully_connected)
and inp_bytes % 16 == 0
and inp_bytes < 8*1024*1024
)On a box with more than two PCIe-only GPUs, the fast path turns itself off. This is a concrete reason TP underperforms there — and a direct argument for pipeline parallelism instead (Module 03). The interconnect isn't a footnote; it's a gate condition in the source.
Making the All-Reduce Faster
Beyond the built-in kernel, vLLM added pluggable high-performance backends. Each targets a different message-size regime.
PyTorch symmetric memory #20759
Two-shot for 2/4 GPUs, multimem for 6/8. Llama-70B: ~7–10% lower TTFT at TP=8 on B200. Later default-on (#24111).
NCCL symmetric memory #24532
Registers buffers so NCCL dispatches symmetric kernels — ~2.0× at 8 MB (micro-bench, TP=8). Wins at large messages.
ROCm QuickReduce (quantized) #19744
Quantizes to INT8/6/4 before reducing — fewer bytes on the wire. MI300 TP=4: TTFT ~2.2× with INT4.
#24532's headline 2× is a micro-benchmark at a favorable size; its end-to-end DeepSeek-R1 gain was ~3.5% req/s. #19744's ~2.2× is partly against a weak RCCL baseline above 16 MB — expected for a technique whose whole point is sending fewer bytes. The direction is right; the exact multiplier is regime-dependent.
Overlapping the All-Reduce with Compute
The deeper move is to stop treating the all-reduce as a blocking barrier. Two compilation passes rewrite the graph so communication hides behind computation.
- Step 1 four GPUs, four partial sums (2, −1, 3, 1)
- Step 2 all-reduce combines them to Σ=5 and broadcasts back
- Step 3 the overlap trick splits it into reduce-scatter + all-gather
- Step 4 async TP overlaps the comm with the next matmul
Use Next step; drag to orbit.
The async-TP PR is honest: on 2×A100 with an 8B model it showed little to no gain. Overlap pays off as the collective gets big enough to hide real work behind it. A follow-up (#28672) even gates sequence parallelism by token count — SP wins at large token counts, fused all-reduce+RMSNorm+quant wins at small ones.
Check Your Understanding
In which setup does the custom all-reduce fast path disable itself?
How do the overlap passes restructure the TP all-reduce?
Pipeline Parallelism: Layers Across the Network
One handoff per stage instead of two all-reduces per layer — and a war against the bubble
When the Fabric Is Slow, Split by Layer
Pipeline parallelism (--pipeline-parallel-size) splits the model along its layers and hands activations stage-to-stage over a point-to-point link. It supports uneven splits, and — crucially — it's the recommended choice when GPUs lack fast NVLink (e.g. L40S) or when spanning nodes.
It communicates far less than TP: one activation handoff per stage boundary, versus two all-reduces per layer. That maps it onto the PCIe/network tiers where TP would stall — recall the custom all-reduce disables itself on >2 PCIe GPUs anyway.
The Bubble
Pipelining's classic weakness: while stage 0 works on microbatch n, the later stages sit idle unless you keep them fed. That idle time is the bubble, and vLLM's PP journey is a march against it.
Watch the Bubble Shrink
This is the stages-versus-time grid. Coloured cells are microbatches being worked; grey cells are idle "bubble". Step between the naive schedule and vLLM's packed one and watch the utilization number move.
- Step 1 naive pipeline — diagonal fill, large grey bubbles, low utilization
- Step 2 vLLM V1 schedules ahead and overlaps the token broadcast, packing the grid and lifting utilization
Toggle to compare; drag to orbit.
The PRs That Killed the Bubble
The original PP support (#4412, 2024) established the shape — one scheduler per stage, blocking send/recv, last stage samples. The V1-era gains came from letting the scheduler run ahead of the pipeline.
Only the last stage samples tokens, which blocked async scheduling. The fix broadcasts sampled tokens from the last PP rank to all others directly on the GPU (no CPU round-trip). Qwen3-30B pp=4: 30.8% E2E throughput, 31.8% TPOT.
Decode scheduled every pp_size steps, prefill chunks pipelined, and the sampled-token broadcast moved onto a separate CUDA stream + process group so it overlaps the stage-to-stage P2P. GB200 PP=4: up to 3.17× tok/s, 3.15× TTFT.
Non-final chunked-prefill chunks discard the sampled token anyway, yet earlier ranks stalled waiting on the last. Skip the broadcast in that case. GLM-4.7 pp4tp2: 1392 → 4578 tok/s.
#42187's 3.17× is the 128000/1 case (huge prefill, one output token) on a single GB200 node with prefix caching off; balanced shapes land at 1.24×–2.28×. #38726's ~3.3× uses output-len=1, a near-pure-prefill workload that flatters a prefill-side fix. Expect smaller gains once real decode is in the mix.
Check Your Understanding
Why is pipeline parallelism preferred when GPUs lack NVLink?
How does #42187 stop the sampled-token broadcast from serializing the pipeline?
Wide EP: Expert + Data Parallelism for MoE
Trade a wasteful all-reduce for a token-sparse all-to-all — then pay an alignment tax
Why MoE Breaks the TP Assumption
In a Mixture-of-Experts model only a handful of experts fire per token. Sharding every expert's weights across all GPUs (plain TP) wastes memory and bandwidth. vLLM's answer — Wide EP — pairs two ideas.
Data-parallel attention
Each DP rank owns its own attention, KV cache, and request stream. Attention weights are replicated (TP=1) or TP-sharded within each DP group (TP>1).
Expert-parallel MoE
Experts shard across an expert-parallel group. Enable with --enable-expert-parallel. Each expert lives on some GPU; tokens route to their top-k experts.
The expert-parallel group size is EP_SIZE = TP_SIZE × DP_SIZE, derived automatically. The routing pattern changes character mid-layer: attention is request-based (each rank handles its own sequences), the MoE block is expert-based (tokens scatter to expert owners and gather back).
A Token's Journey Through a Wide-EP Layer
Between attention and the experts sits a dispatch/combine all-to-all — a token permutation across GPUs. Step through it.
- Step 1 each rank holds its own tokens
- Step 2 dispatch all-to-all routes each token to its expert — E2 gets 3, E3 gets 0
- Step 3 the skew (hot expert) that EPLB rebalances
- Step 4 combine gathers results home
Use Next step; drag to orbit.
The Alignment Tax & Load Balancing
The dispatch/combine collective needs every rank to participate. Two problems follow — and each has a mechanism.
When any DP rank has work, idle ranks must run empty forward passes, or the all-to-all has no partner and deadlocks. The DP Coordinator orchestrates this. It's a real cost of MoE serving — and why data parallelism, not raw replication, is the coordination unit.
Real routing sends far more tokens to some experts than others, so GPUs holding hot experts become stragglers. The Expert Parallel Load Balancer (--enable-eplb) collects routing stats every forward pass and periodically rebalances placement, replicating hot experts across GPUs. Tunable via --eplb-config (window_size 1000, step_interval 3000).
Making Dispatch/Combine Faster
Replaces the naive all-to-all with allgather-for-dispatch, reduce-scatter-for-combine. 4-GPU DeepSeek-R1-FP4: ~1.97× token throughput, TPOT 228 → 116 ms. CUDA-graph compatible.
Quantizes to FP8 before dispatch. MI300X+CX7 DeepSeek-R1: EP=8 → 1.33×, EP=16 → 2.68×. The gain grows with scale because naive all-to-all degrades so badly at EP=16.
One-sided RDMA READ moves expert weights during rebalancing with no staging buffers. DeepSeek-V3.2 EP=8: transfer 0.9 → 0.7s, staging memory 4.4 GiB → 0.
The EP kernel landscape churns fast. The PPLX backend was removed in early 2026 (#33724), and vLLM now carries several all-to-all backends (allgather_reducescatter, DeepEP, FlashInfer, NIXL, MoRI). Treat any "backend X beats Y" claim as version-stamped.
Check Your Understanding
What collective does Wide EP use in the MoE block instead of TP's all-reduce?
What must idle DP ranks do when another rank has a live request?
Disaggregated Serving & KV Transfer
Split prefill from decode across instances — for goodput under an SLO, not raw throughput
Two Phases, Two Machines
Prefill is compute-bound; decode is memory-bound. Disaggregated prefill/decode (P/D) runs them on different vLLM instances, connected by a KV-cache transfer. The counterintuitive part comes straight from vLLM's docs:
What it buys is independent control of the two phases — tune TTFT without disturbing inter-token latency, and reliably bound tail ITL. That improves goodput under a latency SLO. If your only metric is tokens/sec on an unconstrained batch, this is not your tool. If you have a p99 contract, it is.
Watch the KV Cache Move
Step through a disaggregated request: prefill builds the KV blocks, the connector ships them once, decode streams tokens. Two machines, tuned independently.
- Step 1 two instances, two roles
- Step 2 prefill builds the KV cache (tuned for TTFT)
- Step 3 the connector ships it once, prefill→decode
- Step 4 decode streams tokens (tuned for ITL)
Use Next step; drag to orbit.
Three Abstractions, One Directory
All the machinery lives under vllm/distributed/kv_transfer, built from three pieces. The connector is the pluggable part.
send_tensor / recv_tensor.
The Connector Ecosystem
The foundation everything else plugs into. Per-process connectors, layer-wise async transfer, xPyD orchestration deliberately left to external infra. Mechanism, not benchmarks.
The direct high-performance transport: fully async send/recv over NVIDIA's NIXL (UCX/GDS backends), a MultiConnector so several coexist. Supports xPyD and TP>1.
CPU KV offload + cross-instance KV pooling + P/D over NIXL. 2×H100, Llama-3.1-8B, 1P1D: ~40% higher tok/s, ~8× better tail ITL.
A shared-object-store topology so any decode instance can fetch any prefill instance's KV via per-TP-rank key prefixes.
The ~40% / ~8× figures are against a baseline of "two separate vLLM instances" rather than a tuned co-located deployment — an unusual comparison, at a single request rate. The direction (disaggregation helps tail ITL) matches the DistServe research framing; the exact multipliers are baseline-dependent.
Check Your Understanding
What is disaggregated prefill/decode actually good for?
Which is the direct async KV transport that other connectors build on?
CUDA Graphs: Killing Launch Overhead
Once the collectives are fast, the Python between them becomes the bottleneck
Why a Per-Rank Trick Matters at Scale
CUDA graphs capture a sequence of kernel launches once and replay them, eliminating per-step CPU launch cost. That's a single-GPU optimization on its face — so why is it in a multi-GPU walkthrough?
In a TP or PP deployment, CPU launch overhead on the critical path is paid by every rank, every step — and a slow rank makes every other rank wait at the next collective. Removing launch overhead is the difference between the fast collectives of the last four modules being the bottleneck (good) or Python dispatch being the bottleneck (waste).
Full vs Piecewise, Decoupled from Compile
The V1 framework (#20059) decoupled graph capture from torch.compile's piecewise compilation using a "nested wrapper" design, exposing a single cudagraph_mode flag.
On A100, #20059 measured CPU launch time of ~56 ms flattened vs ~28 ms piecewise, FA2 output throughput ~5% up, and Triton ITL down up to 38%. Full-graph capture for Cutlass MLA (#22763) added ~6% E2E on DeepSeek-V2-Lite and roughly halved P99 TTFT (1818 → 1002 ms).
Eight Launches → One Replay
The orange blocks are individual CPU kernel launches, fired one by one every step. Step forward to capture them into a single CUDA graph and replay the whole sequence at once.
- Step 1 eight separate CPU launches, each with overhead, fired sequentially (~56 ms)
- Step 2 captured once and replayed as a single op (~28 ms) — in TP/PP this compounds across every rank
Toggle; drag to orbit.
Where the Gains Landed
The framework: nested wrapper + cudagraph_mode, FA2 and FlashInfer support. Per-rank capture that compounds in TP/PP.
+6% E2E, P99 TTFT 1818 → 1002 ms on DeepSeek-V2-Lite (single-GPU decode path).
Full cudagraph by default for mamba/hybrid models: V1 went from ~2.4× slower than V0 to slightly faster (8.38 → 3.40 s). Removes per-step CPU dispatch that dominates small-batch decode.
Check Your Understanding
Why does a per-rank CUDA-graph win matter in multi-GPU serving?
What cost does a CUDA graph eliminate?
torch.compile & Collective Fusion
The compiler is where communication and computation get fused into one kernel
Piecewise Compilation, Attention as Boundary
vLLM's torch.compile integration uses attention ops as graph-split boundaries so each segment captures cleanly as a cudagraph while attention runs eagerly between them. The compile levels landed in #9715; #10528 made CUDA graphs default-on in V1.
#10273 made custom Inductor passes picklable so the code cache actually works; #11614 keyed the compile cache on traced source including distributed ops, so a change to communication code correctly invalidates compiled artifacts. Boring, essential, and multi-GPU-relevant because the traced set includes the collective code paths.
The Payoff: Fusing the Collective Itself
The compiler's most striking multi-GPU trick is folding a communication collective and its neighbors into one kernel. #21069 detects the tensor-parallel all-reduce → RMSNorm → quant sequence in the FX graph and replaces it with a single FlashInfer op.
x = all_reduce(x) # collective
x = rms_norm(x, w) # elementwise
x = quant_fp8(x) # elementwise- Step 1 three kernels run back-to-back on the TP path
- Step 2 the compile pass fuses them into one FlashInfer op — fewer launches, less memory traffic
Toggle; drag to orbit.
On B200 TP=2, #21069 shows ~7–8% TPOT for FP8 — but the gain is against custom ops. Against default torch.compiled ops, FP8 TTFT can actually regress up to 8%. NVFP4 fares better (5% TPOT/TTFT vs compiled). "Fusion is faster" is true only relative to the right baseline.
Chunked Prefill: The Scheduler-Level Multiplier
Alongside the compiler sits a scheduler trick that makes mixed batching work at all. Chunked prefill splits long prefills so they interleave with decode and bound peak activation memory.
Split long prefills by a token budget so up to two chunked prefills run at once, mixed with decode.
Turned on by default above 32K tokens to bound peak activation memory during profiling.
long_prefill_token_threshold lets multiple long prefills batch concurrently. This mixed batching is exactly what the PP bubble-removal work (Module 03) exploits.
Chunked prefill's multi-GPU relevance is indirect — it shapes batch composition and memory pressure, not cross-device comms. But it's the reason mixed prefill/decode batching works, which the pipeline scheduler depends on.
Check Your Understanding
What does the collective-fusion pass (#21069) fuse together?
Why does chunked prefill matter to the pipeline-parallel work in Module 03?
Choosing a Configuration
The decision is a function of your interconnect and your workload
The Decision Table
Everything in this walkthrough collapses into a handful of choices, driven — as Module 01 promised — by the fabric you have and the metric you're optimizing.
Single node, fast NVLink, dense model
Tensor parallelism (TP = GPUs/node). All-reduce is cheap on NVLink; simplest scaling.
No NVLink, or spanning nodes
Pipeline parallelism (add PP = #nodes). One handoff per stage beats two all-reduces per layer.
Large MoE model
Wide EP (--enable-expert-parallel + --enable-eplb). Sparse all-to-all + hot-expert rebalancing.
A p99 latency / TTFT SLO
Disaggregated P/D (NIXL or LMCache). Tune phases independently; control tail ITL. Won't raise raw throughput.
Any of the above
CUDA graphs + torch.compile (default-on in V1). Per-rank launch-overhead removal compounds across ranks.
They Compose
A production DeepSeek-scale deployment might run every dimension at once — each targeting a different tier of the Module 01 hierarchy.
vllm serve deepseek-ai/DeepSeek-V3 \
--tensor-parallel-size 8 \ # TP within a node (NVLink)
--pipeline-parallel-size 2 \ # PP across nodes (network)
--enable-expert-parallel \ # EP for the MoE layers
--enable-eplb \ # rebalance hot experts
--kv-transfer-config ... # NIXL connector for disaggvLLM's multi-GPU performance work is overwhelmingly communication engineering: faster collectives, cheaper topologies, moving the KV cache exactly once, and clearing the Python and launch overhead that sits between the collectives. The FLOPs were never the hard part.
Final Check
Your 4 GPUs are PCIe-only (no NVLink). First reach for…?
Your binding constraint is a strict p99 inter-token-latency SLO. Which technique targets it directly?
In one phrase, what is vLLM's multi-GPU performance work mostly about?
Keep Going
Every mechanism in this course is grounded in a real merged vllm-project/vllm pull request — with measured speedups and honest caveats on each benchmark. The PR numbers cited throughout (all-reduce backends #20759/#24532, overlap #16155/#17882, pipeline #32618/#42187, expert-parallel #23964/#28664, KV connectors #15960/#17751/#16625, CUDA graphs #20059/#22763, fusion #21069) are the primary sources.
Browse the merged PRs directly on the vLLM repository, and pair this with the other Learn AI courses on attention and GPU kernels.