MPI_Reduce与MPI_Gather+根节点归约在CRAY超算上的性能对比问询

Great question! The short answer is no — MPI_Reduce isn't always faster than the "MPI_Gather + root-node parallel reduction" approach for any N, especially on a CRAY supercomputer with MPICH2 and 32-core nodes. Let's break down why, based on different N values:

When N is much larger than 32 (e.g., N=1024, 4096) #

• MPI_Reduce will almost always be faster. Here's why:
  • Communication overhead: MPI_Reduce uses a tree-based collective algorithm under the hood (optimized for CRAY's high-speed interconnects like Aries/Gemini). This means data is aggregated across nodes in log₂(N) rounds, with each round only passing small chunks of data (single floats, in your case). In contrast, MPI_Gather forces every node to send its float directly to the root, resulting in N individual data transfers to the root — this saturates the root node's network link and adds far more total communication latency/bandwidth usage.
  • Computation distribution: MPI_Reduce spreads the reduction work across all nodes during the tree aggregation. Every node participates in combining partial results, so no single node bears the full computational load. With the "Gather + parallel reduce" approach, all computation falls on the root node's 32 threads — even with parallelism, reducing N floats (where N >>32) will take significantly longer than distributed computation.

When N is small (e.g., N ≤32) #

• The "MPI_Gather + root-node parallel reduce" approach might outperform MPI_Reduce:
  • Communication efficiency: For small N, gathering all floats to the root is a single round of communication. The total data transferred is tiny (N×4 bytes), so this completes extremely quickly on CRAY's low-latency network. MPI_Reduce, by contrast, uses a tree algorithm which requires log₂(N) rounds of communication — each round adds small latency overheads that can stack up to be slower than a single Gather operation.
  • Root-node computation speed: If your MPI_Reduce implementation runs the final reduction on the root node serially (which is common for standard MPI collective operations, since MPI is process-focused, not thread-focused), reducing N floats (≤32) with 32 parallel threads will be nearly instantaneous. The serial reduction in MPI_Reduce can't compete with that level of parallelism for small datasets.

When N is in the middle (32 < N < ~100) #

• The result depends on your specific CRAY system's network characteristics (latency vs. bandwidth) and how MPICH2 is optimized. You'll need to run benchmark tests to compare the two approaches here — factors like the root node's network contention and thread scheduling overhead can swing the performance either way.

Additional considerations #

• MPICH2 on CRAY is heavily optimized for the hardware, so MPI_Reduce might use specialized algorithms (like linear reductions for very small N) that narrow the gap. But even with that optimization, the parallel thread-based reduction on the root can still win for small N.
• Float is a small data type, so latency (not bandwidth) dominates communication costs here. That's why the round count in MPI_Reduce matters more for small N than total data transferred.

内容的提问来源于stack exchange，提问作者wiowou