MPI_Neighbor_alltoallw()中正确的发送接收顺序是什么？

Great question—this is a super common sticking point when setting up ghost cell synchronization with MPI's neighbor collectives. Let's break this down clearly, especially since you're working with custom MPI datatypes and VTK visualization (nice touch, by the way!).

First, let's get a key point out of the way: MPI_Neighbor_alltoallw() doesn't require you to manually specify a "send first, receive later" or vice versa order. It's a collective operation, meaning MPI handles all the underlying synchronization and communication sequencing between neighbors automatically. Your job is to correctly map what you send to each neighbor, and where you receive their data into your ghost cells.

Core Logic for Ghost Cell Scenarios #

For ghost cell updates, each process needs to:

1. Send its own boundary cells to adjacent neighbors (these become the neighbors' ghost cells)
2. Receive boundary cells from adjacent neighbors (these fill its own ghost cells)

The critical part is aligning these two actions to the same neighbor index in your parameter arrays. Here's how to structure this step-by-step:

1. Align Your Neighbor Topology #

First, make sure your MPI topology (e.g., Cartesian grid) correctly defines your neighbors in a consistent order. For example, in a 2D grid, you might fix the order as [North, South, East, West] for all processes. Boundary processes will have MPI_PROC_NULL for missing neighbors—we'll handle those automatically.

2. Configure Send Parameters (Your Boundary → Neighbor's Ghosts) #

For each neighbor index (matching your topology order), define:

• sendbuf: Pointer to your local data buffer containing boundary cells
• sendcounts[]: Number of data elements to send to each neighbor (e.g., your north boundary cell count)
• sdispls[]: Byte offset from sendbuf to the start of the boundary data for that neighbor (use MPI_Aint for this—critical for custom datatypes!)
• sendtypes[]: Custom MPI datatype for the data being sent to each neighbor (can reuse the same type if all boundaries use the same structure)

3. Configure Receive Parameters (Neighbor's Boundary → Your Ghosts) #

For the exact same neighbor index, define:

• recvbuf: Pointer to your local buffer that includes ghost cell regions
• recvcounts[]: Number of elements to receive from each neighbor (should match the sendcounts value the neighbor uses for you)
• rdispls[]: Byte offset from recvbuf to the start of the ghost cell region that this neighbor's data will fill
• recvtypes[]: Custom MPI datatype for the data being received (matches the neighbor's sendtypes for you)

Example Snippet (2D Grid) #

Let's say you're using a 2D grid with row-major memory layout, and ghost cells surrounding your local grid. Here's a simplified code outline:

// Fixed neighbor order: [North, South, East, West]
int num_neighbors = 4;
int sendcounts[4];
MPI_Aint sdispls[4];
MPI_Datatype send_types[4];

// Populate send parameters (your boundaries)
sendcounts[0] = LOCAL_COLS; // North boundary: 1 row of columns
sdispls[0] = (LOCAL_ROWS - 1) * LOCAL_COLS * sizeof(CellData); // Byte offset to last row
send_types[0] = CELL_DATA_TYPE; // Your custom committed datatype

sendcounts[1] = LOCAL_COLS; // South boundary: first row
sdispls[1] = 0;
send_types[1] = CELL_DATA_TYPE;

// ... repeat for East/West boundaries ...

// Populate receive parameters (your ghost cells)
int recvcounts[4];
MPI_Aint rdispls[4];
MPI_Datatype recv_types[4];

recvcounts[0] = LOCAL_COLS; // North ghost: fills the row above your local grid
rdispls[0] = (LOCAL_ROWS + GHOST_SIZE) * (LOCAL_COLS + 2*GHOST_SIZE) * sizeof(CellData);
recv_types[0] = CELL_DATA_TYPE;

recvcounts[1] = LOCAL_COLS; // South ghost: fills the row below your local grid
rdispls[1] = 0;
recv_types[1] = CELL_DATA_TYPE;

// ... repeat for East/West ghosts ...

// Handle boundary processes (MPI_PROC_NULL)
// MPI automatically ignores entries where the neighbor is MPI_PROC_NULL if counts are 0
// So for a north boundary process, set sendcounts[0] = 0 and recvcounts[0] = 0

// Execute the collective
MPI_Neighbor_alltoallw(send_buffer, sendcounts, sdispls, send_types,
                       recv_buffer, recvcounts, rdispls, recv_types,
                       mpi_cart_comm);

Key Gotchas to Avoid #

• Neighbor Index Alignment: Never mix up the order of neighbors between send and receive parameters. If index 0 is North in send arrays, it must be North in receive arrays too.
• Byte Offsets: sdispls and rdispls are in bytes, not elements. This is especially important with custom datatypes—use MPI_Aint and calculate offsets based on the size of your data structure.
• MPI_PROC_NULL Handling: For missing neighbors, set the corresponding sendcounts and recvcounts to 0. MPI will skip communication for those entries without errors.
• Datatype Commit: Make sure your custom MPI datatypes are fully committed with MPI_Type_commit() before using them in the collective call.

Final Note #

You don't need to worry about the actual execution order of sends/receives between neighbors. MPI handles all the synchronization and routing under the hood—your only job is to correctly map which data goes where for each neighbor pair.

Once this is set up, your ghost cells will be properly updated, and your VTK writer should output consistent parallel data for visualization.

内容的提问来源于stack exchange，提问作者Ali