NVLink vs PCIe in Practice: Bandwidth and LLM Serving
Peer-to-peer and NCCL bandwidth measured on an RTX A6000 and an A100 server, and what NVLink and PCIe change when vLLM serves Qwen3.8-27B with tensor parallelism.
- Doc no.
- KKZ-TN-153
- Date
- 2026-11-02
- GMT day
- 2026:306
- Est. read
- 23 min
GPU Server Topology: NVLink, PCIe and NUMA read the topology of two servers. This post measures them: how fast each path between two GPUs moves data, and how much that matters when the server runs an LLM.
- Machine 1: 2 × RTX A6000 48 GB in a 2 × Xeon Gold 6248R server, one GPU per socket. The GPUs only meet over PCIe 3.0 and UPI (
SYS). - Machine 2: 4 × A100 80 GB PCIe in a 2 × Xeon Platinum 8368 server. GPU0–1 and GPU2–3 are NVLink bridge pairs (
NV12), one pair per socket; every other pair isSYS.
The model is Qwen3.8-27B with INT8 weights and BF16 activations (W8A16), served by vLLM 0.32 with tensor parallelism (TP). Each data point is a single run with 4 to 256 requests, so differences of a few percent are within noise.
01Background#
1.1Tensor Parallelism#
A model can be split across two GPUs in more than one way. Tensor parallelism (TP), which vLLM uses here, splits every weight matrix in half, so each GPU stores half the weights and does half the work of every layer. The halves are not independent: each GPU ends a layer with only part of the answer, and the two have to combine their parts before either can go on.
Take the MLP of one layer, , at TP=2:
- is split by columns. Each column of gives one entry of the intermediate vector , so with half the columns, each GPU computes half of on its own, without communication.
is split by rows, to match: each GPU holds the rows that multiply its half of . Each GPU computes a full-size vector, but only part of the sum:
- An all-reduce adds the two parts and leaves the sum on both GPUs, so both start the next layer with the same .
Attention is split the same way, by heads: each GPU computes half the heads, and the output projection, split by rows, again leaves two partial sums. Qwen3.8-27B mixes two kinds of attention layer, full attention and Gated DeltaNet (GDN), a linear attention that keeps a fixed-size state per sequence instead of per-token KV; both split by heads. So every layer ends in two all-reduces, one after attention and one after the MLP, and the next layer waits for each.
1.2All-Reduce Size#
Each all-reduce carries the layer’s output for every token in the step: tokens × hidden size × 2 bytes in BF16. Qwen3.8-27B has a hidden size of 5120, so:
- one decode step of one sequence all-reduces 10 KiB, twice per layer;
- one prefill chunk of 8192 tokens (vLLM’s
max_num_batched_tokenshere) all-reduces 80 MiB, twice per layer.
Decode sends many tiny messages; prefill sends a few large ones. How long they take depends on how the bytes get from one GPU to the other.
1.3NCCL#
In every run here, the all-reduces go through NCCL, which moves data between two GPUs in one of two ways:
- P2P: a GPU reads or writes the other GPU’s memory directly, over NVLink or PCIe.
- SHM: the sender copies into a buffer in host memory, and the receiver copies it out.
Neither uses the CPU cores: NCCL kernels on the GPUs move the data themselves. The two differ in which parts of the CPU the data passes through.
P2P never touches a memory controller or DRAM. A peer write leaves GPU0 over PCIe, enters CPU0 at a host bridge, crosses the on-die fabric and UPI to CPU1, and leaves through CPU1′s host bridge into GPU1. The CPUs only route it; on the A6000 server, Intel PCM saw little DRAM traffic during a peer copy (below). Over NVLink, P2P does not pass through the CPUs at all.
SHM goes into DRAM and back out. GPU0 writes into a buffer through a memory controller, and GPU1 reads it back. The data crosses PCIe twice, out of GPU0 and into GPU1, plus a round trip to DRAM, so SHM is slower. It works wherever the GPUs can reach host memory, which is why NCCL falls back to it.
NCCL uses P2P only for pairs no farther apart than NCCL_P2P_LEVEL. Its distance ladder is close to the codes of nvidia-smi topo -m, but without NODE: any path through one CPU counts as PHB. When the variable is unset, NCCL’s source (ncclTopoCheckP2p in paths.cc) uses PXB, so P2P stops at PCIe switches. The one exception is an AMD CPU with at most two GPUs, which gets SYS.
Both machines here have Intel CPUs, so pairs on different sockets fall back to SHM. Raising the level from PXB to SYS lets them use P2P:
NCCL_P2P_LEVEL cuts NCCL’s distance ladder. The default allows P2P up to PXB; SYS allows every path. The NVLink pair uses P2P either way.NCCL’s debug log names the transport:
| Pair | Default | NCCL_P2P_LEVEL=SYS |
|---|---|---|
| NVLink pair | P2P/CUMEM/read | P2P/CUMEM/read |
| Across sockets | SHM/direct/direct | P2P/CUMEM |
PIX and PXB behaved like the default on every cross-socket pair.
1.3.1Link Bandwidth#
A P2P transfer is only as fast as the narrowest link on its path. Each GPU hangs off a PCIe x16 link, rated per direction as:
| Link | Per lane | Rated | A large copy normally reaches |
|---|---|---|---|
| PCIe 3.0 x16 | 8 GT/s | 15.75 GB/s | about 12.6 GB/s |
| PCIe 4.0 x16 | 16 GT/s | 31.5 GB/s | about 25 GB/s |
| PCIe 5.0 x16 | 32 GT/s | 63.0 GB/s | about 50 GB/s |
Sixteen lanes, less the 128b/130b line encoding, give the rated speed. Packet headers and flow control take more, so a large copy normally gets about 80% of it.
The two sockets are joined by several UPI links side by side. Each carries 2 bytes per transfer in each direction:
| CPUs | UPI | Per link, per direction | All links, per direction |
|---|---|---|---|
| 2 × Xeon Gold 6248R (Machine 1) | 2 × 10.4 GT/s | 20.8 GB/s | 41.6 GB/s |
| 2 × Xeon Platinum 8368 (Machine 2) | 3 × 11.2 GT/s | 22.4 GB/s | 67.2 GB/s |
| 2 × Xeon Platinum 8558P | 3 × 20 GT/s | 40.0 GB/s | 120 GB/s |
These are rated speeds; UPI also carries headers, checksums and coherence traffic, so data moves slower. Whether one copy spreads over several links depends on how the CPU routes it, so the figure below draws a single link, the least a copy could have.
On both machines the peer copy falls short of every pipe on its path. Copies between a GPU and host memory show where the shortfall lies. They cross the same PCIe link, and when the memory sits on the other socket, the same UPI:
cudaHostRegister), with the pages and the copying thread bound to one NUMA node; each bar spans the two GPUs measured, median of three runs. The dashed line is 80% of the GPU’s PCIe rate, what a large copy normally reaches.The two machines fail in different places:
- Machine 1: every copy falls well short, host copies included, at 51–55% of PCIe 3.0. Even a GPU and its own socket’s memory, with no UPI on the path, reach only half the link. The limit lies between the GPU and its CPU, not between the sockets; the A6000 section looks further.
- Machine 2: host copies reach 78–83% of PCIe 4.0, which is normal, and the other socket’s memory is as fast as the GPU’s own. A host copy across sockets crosses UPI at about 25 GB/s, so neither the PCIe link nor UPI holds the peer copy to 18.5 GB/s. What is left is the step only P2P takes: the CPU0 host bridge forwarding the GPU’s writes over UPI to a device under CPU1.
1.4Roofline#
A roofline compares two speeds of the GPU, peak compute and memory bandwidth. Their ratio is the ridge point, in FLOPs per byte. A kernel that does fewer FLOPs per byte it reads than the ridge waits on memory; one that does more waits on compute.
1.4.1Arithmetic Intensity#
Two facts place every part of a step:
- A linear layer reads its weights once per step and uses them for every token in the step. The weights are INT8 and multiplied in BF16, so each byte gives 2 FLOPs per token; with the BF16 output layer, the whole model gives 1.9 FLOPs per byte per token. More tokens per step move it right.
- Attention reads each sequence’s own KV. A byte of FP8 KV serves 6 query heads, 2 FLOPs each: 12 FLOPs per byte, 48 with MTP’s four tokens. More sequences do not move it.
Decode sits left of both ridges, prefill far right. Only the A100′s concurrency-128 runs cross over.
1.4.2With TP#
TP=2 gives each GPU half of every weight matrix and half the KV heads. Each GPU reads half the bytes and does half the FLOPs, so the FLOPs per byte, and the ridge, stay where they were. What TP adds is the all-reduces:
With a free link TP=2 would be exactly twice as fast. How much of that a real link takes depends on the size of the step:
- A small step sends small all-reduces, about 10 µs each whatever the link (Figure 10), so latency decides.
- A large step, such as a prefill chunk, sends large ones, so bandwidth decides.
For the large case, take one token of a prefill chunk on one GPU of a pair: 25.6 GFLOP at the GPU’s peak, and 128 all-reduces of 10 KiB at the bandwidth nccl-tests measured for large messages:
| Compute | All-reduce | |
|---|---|---|
| RTX A6000, PCIe | 165 µs | 182 µs |
| A100, PCIe | 82 µs | 96 µs |
| A100, NVLink | 82 µs | 8 µs |
The two run one after the other, so over PCIe a prefill chunk takes about twice as long as its compute, and over NVLink about 10% longer. Both times grow with the tokens, so the ratio holds for any chunk size.
The roofline figure of each machine below covers both cases. It times the 128 all-reduces of a B-token step with nccl-tests and adds that time to the GPU’s.
02A100#
This is Machine 2 of GPU Server Topology: NVLink, PCIe and NUMA. Each A100 has a PCIe 4.0 x16 link, 31.5 GB/s per direction on paper, and each bridge pair has 12 NVLinks, 300 GB/s per direction.
All serving runs on this machine:
- use an FP8 KV cache, FlashInfer attention and prefix caching, with MTP off so that draft acceptance does not differ between runs;
- run TP=2 with
NCCL_P2P_LEVEL=SYSand--disable-custom-all-reduce, so that NCCL does all of the communication; - use GPU0–1 as the NVLink pair and GPU1–2, on different sockets, as the PCIe pair;
- have 32 requests of 128 output tokens per point, except TP=2 at concurrency 1, which has 4.
2.1KV Cache Capacity#
max_num_seqs | Weights per GPU | KV memory per GPU | KV cache size | |
|---|---|---|---|---|
| TP=2, NVLink (GPU0–1) | 1 | 14.61 GiB | 52.89 GiB | 3,208,045 |
| TP=2, PCIe (GPU1–2) | 1 | 14.61 GiB | 55.05 GiB | 3,338,690 |
| TP=4 | 16 | 7.37 GiB | 59.97 GiB | 7,273,989 |
TP=4 holds 2.3 times as many tokens as TP=2: it has more free memory per GPU and stores half the bytes per token.
“KV cache size” is the number vLLM prints at startup. For this model it is not a count of token slots, and it changes with max_model_len; vLLM Hybrid KV Cache works out what it counts.
With identical settings, the NVLink pair gets 2.2 GiB less KV memory than the PCIe pair. The cause is not known; NCCL buffers for the NVLink transport are one candidate.
2.2Serving at TP=1 and TP=2#
TP=1 runs on GPU0 alone. Output throughput, prefill included:
max_num_seqs=1; the rest with 8.- TP=2 over NVLink serves 1.51 to 1.75 times as much as TP=1. At every length its lead grows with concurrency, and it is largest at 128K and concurrency 8.
- TP=2 over PCIe serves 1.38 to 1.61 times as much. Up to 16K its lead shrinks with concurrency, from 1.46 to 1.38 at 1K: more sequences make the all-reduces larger, and PCIe pays for each byte. At 64K and 128K, where attention takes most of the step, it grows with concurrency like NVLink’s.
- At concurrency 1 both pairs serve about 1.5 times as much as TP=1, well short of 2, so per GPU a single A100 does more work.
2.3NVLink vs PCIe at TP=2#
At concurrency 1 the two pairs decode at the same speed: within 0.7% at every length, 65.8 against 65.4 tok/s at 128K. NVLink cuts time to first token by 6–26%, and the whole request by 2–5%.
In nccl-tests, a 10 KiB all-reduce, the size of a decode message, takes about 10 µs on either link. An 80 MiB prefill message takes 0.5 ms over NVLink and 6 ms over PCIe:
NCCL_P2P_LEVEL=SYS. Time is in µs on the left and in ms on the right.Below about 64 KiB the time is fixed cost: launch, synchronization and a round trip, which bandwidth cannot shorten. A decode step of one sequence sends only such messages, 128 of them at about 10 µs, some 1.3 ms of a 15 ms token on either link. Prefill sends 80 MiB messages, where NVLink’s bandwidth matters.
More sequences make decode messages larger, and the gap opens. NVLink’s gain in output throughput over PCIe:
max_num_seqs=1, 2–8 with 8, and 16 with 16.At every length the gain grows with concurrency, up to 8–33% at 16. At 16 sequences a decode message is 160 KiB, where PCIe takes about four times as long as NVLink. The gain shrinks at 128K, where attention dominates the step.
TP=4 spans both pairs, so every all-reduce crosses the sockets. At concurrency 16:
| Input | TP=2, PCIe | TP=2, NVLink | TP=4 |
|---|---|---|---|
| 1K | 282.8 tok/s | 377.3 tok/s | 435.0 tok/s |
| 8K | 421.1 tok/s | 513.8 tok/s | 506.1 tok/s |
| 32K | 176.4 tok/s | 216.1 tok/s | 262.5 tok/s |
| 128K | 140.6 tok/s | 151.9 tok/s | 190.8 tok/s |
TP=4 is faster at long inputs, where it splits attention four ways, and no faster at 8K. Per GPU, a TP=2 NVLink pair does better at every length.
2.4Peer-to-Peer and NCCL#
Every pair, with p2p_topo for the all-reduce:
| GPUs | Link | Copy one way | Copy both ways | All-reduce, default | All-reduce, SYS |
|---|---|---|---|---|---|
| 0, 1 | NV12 | 278.6 GB/s | 526.7 GB/s | 171.8 GB/s | 172.0 GB/s |
| 2, 3 | NV12 | 278.7 GB/s | 526.9 GB/s | 171.6 GB/s | 171.8 GB/s |
| 0, 2 | SYS | 18.5 GB/s | 36.4 GB/s | 12.0 GB/s | 12.7 GB/s |
| 0, 3 | SYS | 18.5 GB/s | 37.1 GB/s | 12.1 GB/s | 12.7 GB/s |
| 1, 2 | SYS | 18.4 GB/s | 37.0 GB/s | 12.1 GB/s | 12.7 GB/s |
| 1, 3 | SYS | 18.5 GB/s | 35.5 GB/s | 12.0 GB/s | 12.3 GB/s |
All-reduces are FP32 with 64 MiB messages. In bandwidth NVLink is 15 times faster for copies and 13.5 times for all-reduce.
- NVLink copies at 93% of its 300 GB/s.
- A cross-socket copy reaches 59% of PCIe 4.0, about the same share as the A6000 pair’s 62%. Unlike the A6000, this machine’s host copies reach 78–83% (Figure 5), so the shortfall here is specific to P2P across sockets.
NCCL_P2P_LEVEL=SYSadds 3–6% on cross-socket pairs, much less than the 27% on the A6000, and nothing on NVLink pairs, which use P2P anyway.
nccl-tests agrees on the bandwidth, with two details:
- With CUDA Graphs, which vLLM uses for decode, cross-socket P2P at 64 MiB rises from 12.2–12.5 to 13.1–13.5 GB/s, while NVLink drops from 174 to 148–155 GB/s, for reasons not yet found.
- For small messages,
SYScuts latency across sockets: on GPU1–2, a 40 KiB all-reduce takes 12.7 µs instead of 18.1.
Four GPUs behave differently. Over GPU0–3, the default mixes NVLink P2P inside the pairs with SHM between them, and reaches 34.5 GB/s busbw at 64 MiB. With SYS everything is P2P, and it drops to 32.5 GB/s; a 1 MiB all-reduce takes 155 µs instead of 109. Set it for a cross-socket pair, but measure first on a mixed topology.
2.5Training#
Short TP=2 training runs, 16 steps each, of one 1024-token sequence per step. LoRA uses Qwen3-32B. Full fine-tuning (SFT) and continued pretraining use Qwen3-14B, because the FP16 weights, gradients and AdamW states of 32B need about 122 GiB per GPU at TP=2.
| NVLink, s/step | PCIe, s/step | Data per GPU per step | Average rate | |
|---|---|---|---|---|
| LoRA, 32B | 2.106 | 2.100 | 6.22 GB | 3.0 GB/s |
| SFT, 14B | 1.026 | 1.041 | 3.52 GB | 3.4 GB/s |
| Pretraining, 14B | 1.003 | 1.041 | 3.52 GB | 3.5 GB/s |
NVLink is 0–4% faster. Spread over a step, the traffic needs about 3.5 GB/s, a quarter of what the PCIe pair delivers. This is a small, unoptimized setup (eager mode, activation checkpointing, batch of one); larger batches would move more data per step, but they also compute longer.
2.6Other Workloads#
Qwen3.8-27B with shared prefixes gains at most a third from NVLink. Two more experiments look for a larger gap: prompts that share nothing, so that every request runs its full prefill, and smaller models at high concurrency. Each bar is NVLink’s output throughput divided by PCIe’s, at TP=2:
Distinct prompts widen the gap a little for long inputs: at 8K from 1.23× to 1.37×, at 32K from 1.22× to 1.34×. Each request now runs its full prefill, with its large all-reduces. Absolute throughput falls by a factor of 7 to 14 on both pairs, so most of that time is compute.
Small models at high concurrency widen it much more. Qwen3-4B goes from 1.02× at concurrency 1 to 1.78× at 128, and a repeat with new prompts gave 1.80×. Qwen3-32B reaches only 1.41×.
The difference is the ratio of communication to compute. An all-reduce carries hidden size × 2 bytes per token, while a token costs about 2 × parameters in FLOPs. Per forward pass and token:
| Layers | Hidden size | All-reduced | Compute | |
|---|---|---|---|---|
| Qwen3-4B | 36 | 2560 | 369 KB | 8 GFLOP |
| Qwen3-32B | 64 | 5120 | 1.3 MB | 64 GFLOP |
The small model moves 2.3 times as much data per FLOP. High concurrency then pushes its messages past 64 KiB, where bandwidth matters.
2.7Roofline#
The A100 80 GB PCIe has 312 TFLOPS of dense BF16 and 1,935 GB/s, so its ridge point is 161 FLOPs per byte, which the linear layers reach at about 85 tokens per step. It has twice the A6000′s compute but 2.5 times its bandwidth, so it turns compute-bound at fewer tokens. Only the concurrency-128 runs pass the ridge; the serving sweeps, at 1 to 16 tokens per step, are memory-bound:
A decode step’s floor is 13.9 ms on one GPU and 6.9 ms per GPU at TP=2.
Decode of one sequence is latency-bound on both links. 128 all-reduces of 10 KiB take 1.0 ms over NVLink and 1.1 ms over PCIe, about 13% of a step at the floor. Two things follow:
- The two pairs decoded at the same speed at concurrency 1.
- Even over NVLink, TP=2 decodes at most 1.75 times as fast as one A100. Measured, it serves 1.51–1.52 times as much as TP=1 at concurrency 1 (Figure 9).
With more tokens per step, PCIe becomes bandwidth-bound. At 16 tokens the messages are 160 KiB:
| At 16 tokens per step | NVLink | PCIe |
|---|---|---|
| All-reduces | 1.15 ms | 4.6 ms |
| Step floor | 8.1 ms | 11.5 ms |
The floors differ by a factor of 1.42, but the measured gain at concurrency 16 was smaller, 8–33%. Real steps run well above the floor, 25 ms at 8K over NVLink, and the link’s extra 3.4 ms is a smaller share of them.
In prefill, PCIe takes longer than the compute. Per token:
- all-reduces: 8 µs over NVLink, 96 µs over PCIe;
- compute: 82 µs.
For 8192-token chunks the PCIe link alone allows 10,300 tok/s, against a compute ceiling of 12,200. The two run one after the other, so the PCIe pair prefills at about 5,600 tok/s, against 11,100 over NVLink: less than one A100 at its peak.
The distinct-prompt runs test this. The model says PCIe costs 89 µs per token more than NVLink. With 32 requests at concurrency 16:
| Input | Tokens prefilled | PCIe’s extra time, model | PCIe’s extra time, measured |
|---|---|---|---|
| 8K | 262K | 23 s | 20.7 s (77.0 s against 56.3 s) |
| 32K | 1.05M | 93 s | 79 s |
So the link costs about what the roofline predicts. The throughput ratio is 1.37, not 2, because compute ran far below its roof: the NVLink pair prefilled at about 4,700 tok/s, under 40% of its ceiling, so the same 89 µs is a smaller share of each token.
03RTX A6000#
This is Machine 1 of GPU Server Topology: NVLink, PCIe and NUMA. Each GPU has a PCIe 3.0 x16 link, 15.75 GB/s per direction on paper, and a transfer between them crosses PCIe, CPU0, UPI, CPU1 and PCIe again.
All serving runs on this machine:
- use an FP8 KV cache, FlashInfer attention, prefix caching, MTP with 3 draft tokens and
max_num_seqs=8; - run TP=2 with
NCCL_P2P_LEVEL=SYSand--disable-custom-all-reduce: vLLM’s own all-reduce hung at startup on this machine, so NCCL does all of the communication; - use synthetic prompts, one per length, so after a warmup request its prefix is in the cache;
- have 8 requests of 128 output tokens per point, at temperature 0.
Throughput is output tokens divided by wall time, prefill included.
3.1KV Cache Capacity#
| Weights per GPU | KV memory per GPU | KV cache size | |
|---|---|---|---|
| TP=1 | 29.68 GiB | 10.64 GiB | 276,323 |
| TP=2 | 15.09 GiB | 25.65 GiB | 1,331,743 |
TP=2 has 4.8 times the capacity: each GPU has 2.4 times the free memory, and holds only half of each token’s KV and each sequence’s GDN state. TP=1 fits two 128K requests (2.09×), which matters below.
3.2Serving at TP=1 and TP=2#
TP=2 decodes 1.33 to 1.66 times as fast as TP=1 at every point:
- The lead grows with the input, from about 1.4 times at 1K to 1.6 at 128K. Each GPU reads half of every sequence’s KV, and long inputs make attention a larger share of the step.
- Both are slower at 1K, where MTP accepts 2.7 draft tokens per step instead of 3.5–3.9.
- At 128K and concurrency 8, TP=1 runs out of KV and preempts a request. The figure leaves out the recompute, but output throughput falls to 6 tok/s.
In output throughput, prefill included, TP=2 serves 1.3 to 1.6 times as much as TP=1 on twice the GPUs. Per GPU, a single card does more work.
The usual per-request decode speed, output tokens after the first divided by the time after the first token, counts waiting as decoding. With chunked prefill, a decode step that shares the GPU with another request’s prefill chunk lasts as long as that chunk. At 128K and concurrency 2:
- On TP=2, one request got its first token at 3.4 s, then waited 2.7 s while the other prefilled.
- On TP=1, both requests prefilled in the same steps and never waited.
- By the usual measure, TP=1 came out ahead: 52.2 against 44.7 tok/s.
So the figure leaves out every gap between tokens longer than 0.2 s. A normal step took at most 164 ms and a stall at least 0.53 s, so any threshold in between gives the same numbers.
3.3Profiling#
Nsight Systems traced TP=2 at 128K input and 1024 output tokens. The numbers below cover 80 steady decode steps at each concurrency, averaged over the two GPUs. Each step also verifies MTP drafts, so it covers 4 tokens of every sequence.
- Linear layers: about 30 ms at any concurrency. A step reads the weights once, whatever the batch.
- Full attention: grows from 8 to 33 ms, because each sequence reads its own 128K tokens of KV.
- NCCL: grows from 5.6 to 13.1 ms. Every step runs 138 all-reduces and 7 all-gathers, and the mean all-reduce rises from 32–40 µs at concurrency 2 to 74–82 µs at 8.
NCCL is 11–15% of the step and never overlaps compute, so removing it entirely would make concurrency 8 at most 1.17 times faster. On this machine, attention costs more than PCIe.
3.4Peer-to-Peer and NCCL#
A large copy over PCIe 3.0 x16 normally reaches about 12.6 GB/s. On this machine, the peer copy runs at 9.75 GB/s from GPU0 to GPU1 and 10.25 GB/s back, 62–65% of the 15.75 GB/s rate, and 19.24 GB/s with both directions at once. Copies to and from host memory get only 8.0–8.7 GB/s (Figure 5).
Even a copy between a GPU and its own socket’s memory gets only about half the link rate, less than the copy across sockets. Intel PCM, sampled during the peer copy, rules out the obvious causes:
- both links stayed at PCIe 3.0 x16;
- UPI ran at about 42% of its capacity;
- the target root port wrote about 9.8 GB/s, the same as the copy;
- host memory traffic stayed small, so the data was not staged in DRAM.
So the result is odd, and what holds every copy on this machine to 50–65% of the link is still an open question.
| All-reduce | SHM (default) | P2P (NCCL_P2P_LEVEL=SYS) |
|---|---|---|
p2p_topo, FP32, 64 MiB | 5.54 GB/s | 7.04 GB/s |
| nccl-tests, BF16, 128 MiB | 5.59 GB/s | 7.03 GB/s |
| nccl-tests, 10 KiB, CUDA Graph | 14.5 µs | 9.4 µs |
| nccl-tests, 40 KiB, CUDA Graph | 23.3 µs | 17.3 µs |
P2P gives 27% more bandwidth, and a quarter to a third less latency for small messages. In serving, with MTP off, a BF16 KV cache and one request at a time, setting NCCL_P2P_LEVEL=SYS on TP=2:
- cut time to first token by 7–10% at every length;
- raised decode speed by 2.4–3.1%;
- raised output throughput by 3.9–8.8%.
Most of the gain is in prefill, where the messages are large.
3.5Roofline#
The RTX A6000 has 154.8 TFLOPS of dense BF16 and 768 GB/s of memory bandwidth. Its ridge point is 202 FLOPs per byte. At 1.9 FLOPs per byte per token, Qwen3.8-27B’s linear layers reach it at about 106 tokens per step: below that, a step takes as long as reading the weights, however many tokens it holds. The serving runs above put 8 to 32 tokens in a step, concurrency 2 to 8 with 4 tokens of each sequence, so decode is memory-bound:
A decode step reads the weights:
- at TP=1, 26.9 GB, at least 35.0 ms;
- at TP=2, 13.4 GB per GPU, at least 17.5 ms.
The link takes back part of that halving: 128 all-reduces of 10 KiB take 1.2 ms over this PCIe pair, and at 32 tokens, concurrency 8 with MTP, 7.9 ms.
In prefill the link is as slow as the compute: per token, 182 µs of all-reduce against 165 µs of compute. For 8192-token chunks the link alone allows 5,500 tok/s against a compute ceiling of 6,000 for the pair, so TP=2 prefills at about 2,900 tok/s, less than one A6000 at its peak.
Adding the KV each step reads, the model gives TP=2 over TP=1 for the steady decode speeds of Figure 14:
| Concurrency 2 | 4 | 8 | |
|---|---|---|---|
| 2K, model | 1.67 | 1.52 | 1.39 |
| 2K, measured | 1.45 | 1.40 | 1.34 |
| 128K, model | 1.73 | 1.67 | 1.67 |
| 128K, measured | 1.61 | 1.66 | 1.62 |
The model follows the measured trend.
- At short inputs the gain shrinks with concurrency, because the weights take the same time every step while the all-reduces grow.
- At 128K the KV reads, which TP halves, hold it near 1.65.
- At 2K the model runs high: it leaves out costs that do not halve, and in the trace NCCL took 1.6 times the model’s estimate, 13.1 ms per step against 7.9 at concurrency 8.
04Summary#
- At low concurrency, TP over PCIe costs little. Decode all-reduces are small and take about 10 µs on any link. NVLink helps time to first token, not decode speed.
- NVLink pays off with larger batches and prefill-heavy traffic: 8–33% more throughput for Qwen3.8-27B at concurrency 16, and up to 1.8 times for a 4B model at 128.
- A roofline with the link added explains most of this. Decode is memory-bound and its all-reduces latency-bound, about 1 ms per step on any link. Prefill is compute-bound, and there PCIe’s all-reduces take about as long as the compute.
- On the A6000 server, NCCL was 11–15% of a decode step; full attention over long contexts cost more.
- On a two-GPU cross-socket pair,
NCCL_P2P_LEVEL=SYSis worth setting once P2P is known to work. On four GPUs, measure first.
- NCCL
The NVIDIA Collective Communications Library, pronounced “nickel”. It is a library, not a network protocol.
It implements collectives such as all-reduce, all-gather and broadcast, in which a group of GPUs exchange and combine data, plus point-to-point send and receive.
For each pair of GPUs, NCCL picks a transport: NVLink or PCIe P2P, host shared memory, or the network through InfiniBand, RoCE or TCP sockets. It also picks an algorithm such as ring or tree, then runs the transfer as CUDA kernels on the GPUs.
PyTorch’s
ncclbackend fortorch.distributed, and through it vLLM, call NCCL for multi-GPU communication. It is open source. ↩