Hey there, let's break down your questions one by one based on common multi-GPU training pitfalls I've run into over the years:

1. Why is backpropagation taking so long, and is this normal? #

This isn’t normal if you expected a meaningful speedup with 2 GPUs. The most likely reasons are:

• Batch size didn’t scale: If you kept the total batch size identical to single-GPU training (instead of doubling it), each GPU is processing the same volume of data as before—so total runtime won’t change. Multi-GPU training only delivers gains when you can proportionally increase your total batch size.
• Suboptimal parallelization strategy: If you’re using DataParallel instead of DistributedDataParallel (DDP), the overhead of syncing gradients to a single main GPU can cripple performance. DDP uses ring-allreduce, a far more efficient sync mechanism that spreads communication load across all GPUs.
• Gradient sync bottleneck: Even with DDP, if your model has an enormous number of parameters, the all-reduce step during backprop can become a choke point. This is especially true if your GPUs are connected via slow PCIe (instead of NVLink), as sharing gradients across GPUs will eat into your speed gains.

2. Forward pass seems fine, but why is the brown region sparse while the gray region shows GPU activity? #

That sparse brown region points to load imbalance or unoptimized computation/transfer overlap:

• The gray region tracks overall GPU utilization, but the brown region might be focused on a specific phase of the forward pass (like computation for a subset of layers). If some layers can’t be parallelized (e.g., custom layers that aren’t GPU-aware, or global operations that can’t be split across GPUs), one GPU might finish its work early while the other is still computing—creating gaps in that specific phase’s activity.
• It could also mean your data loading pipeline isn’t feeding GPUs fast enough. If the forward pass idles waiting for inputs, you’ll see sparse activity in certain stages even if the GPU is busy overall.
• If you’re using a kernel-level profiler, the brown region might show gaps between kernel launches. Small layers can cause this: launching kernels has overhead, and tiny compute tasks might not fill the GPU’s capacity fully.

3. Why are there so many peer-to-peer (P2P) memcpy operations after the forward pass, even with few BN layers? #

Peer-to-peer memcpy is direct memory transfer between GPUs (bypassing the CPU)—here’s why you’re seeing so much of it:

• DataParallel’s output gathering: If you’re using DataParallel, after the forward pass, all GPU outputs are copied to the main GPU for loss calculation. This triggers massive P2P transfers, especially if your output tensors are large.
• Distributed sync overhead: Even with DDP, some tensor sync might happen post-forward pass. For example, if you’re using sync_batchnorm (even with few BN layers), syncing running stats across GPUs requires P2P transfers.
• Imbalanced tensor distribution: If your model splits data unevenly across GPUs, or if certain layers produce tensors that need aggregation across GPUs before backprop, you’ll see P2P copies. Attention layers needing global context, for example, might require transferring intermediate tensors between GPUs.
• Unoptimized P2P settings: First confirm your GPUs support P2P communication (check with nvidia-smi topo -m). Even if enabled, some operations might default to transfers if your parallelization logic isn’t tuned to minimize cross-GPU data movement.

The first fix I’d recommend is switching to DDP, paired with scaling your batch size and using torch.profiler to pinpoint exactly which operations are triggering those P2P transfers.

内容的提问来源于stack exchange，提问作者Kapok