MPI3共享内存访问同步：该代码是否符合MPI标准要求？

The Problem #

I've introduced MPI-3's shared memory mechanism into my code—processes sharing this memory can read/write directly without calling MPI library functions. While I've seen examples of one-sided communication using shared or non-shared memory, I couldn't find resources on how to properly access shared memory directly. I wrote the code below, which runs fine on my x86 laptop and server, but I want to know if the MPI standard guarantees it will always work?

My Code #

// Initialization:
MPI_Comm comm_shared;
MPI_Comm_split_type(MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, i_mpi, MPI_INFO_NULL, &comm_shared);

// Memory allocation
const int N_WIN=10;
const int mem_size = 1000*1000;
double* mem[10];
MPI_Win win[N_WIN];
for (int i=0; i<N_WIN; i++) {
    // Need multiple buffers
    MPI_Win_allocate_shared( mem_size, sizeof(double), MPI_INFO_NULL, comm_shared, &mem[i], &win[i] );
    MPI_Win_lock_all(0, win[i]);
}

while(1) {
    MPI_Barrier(comm_shared);
    ... // Write to arbitrary positions in shared memory
    MPI_Barrier(comm_shared);
    ... // Read shared memory content written by other processes
}

// Cleanup
for (int i=0; i<N_WIN; i++) {
    MPI_Win_unlock_all(win[i]);
    MPI_Win_free(&win[i]);
}

I use MPI_Barrier() to ensure synchronization, and assume hardware guarantees consistent memory views. Also, since I'm using multiple shared windows, a single MPI_Barrier seems more efficient than calling MPI_Win_fence() on each shared memory window.

Questions:

1. Is this a legal/correct MPI program?
2. Are there more efficient implementations?

Answers #

1. Legality & Correctness #

Your code is fully compliant with the MPI-3 standard—here's the breakdown:

• Shared Memory Domain Setup: MPI_Comm_split_type with MPI_COMM_TYPE_SHARED correctly groups processes that share a physical memory domain, which is a prerequisite for direct access via MPI_Win_allocate_shared. Only processes in comm_shared can access the allocated shared memory, and your code strictly follows this rule.
• Direct Access Permissions: MPI-3 explicitly permits direct read/write operations on memory allocated via MPI_Win_allocate_shared, provided proper synchronization is in place.
• Synchronization Validity: Using MPI_Barrier(comm_shared) to separate read/write phases is valid. MPI barriers guarantee that all memory operations (including direct shared memory accesses) completed before the barrier are visible to all processes after the barrier. Pairing this with MPI_Win_lock_all(MPI_MODE_NOCHECK) is safe because your barrier-separated phases prevent conflicting concurrent accesses, which aligns with the NOCHECK mode's requirement that you manage access conflicts yourself.

2. Efficiency Optimizations #

Your choice to use a single barrier instead of per-window MPI_Win_fence is a good call, but there are further tweaks to boost performance:

• Merge Multiple Windows into One: Creating 10 separate MPI_Win instances adds unnecessary overhead for MPI's internal window management. Instead, allocate a single large shared window and partition it into logical buffers:

  const int total_mem_size = N_WIN * mem_size;
  double* mem;
  MPI_Win win;
  // Allocate one large shared window
  MPI_Win_allocate_shared(total_mem_size, sizeof(double), MPI_INFO_NULL, comm_shared, &mem, &win);
  MPI_Win_lock_all(0, win);
  
  // Access the i-th buffer via mem[i * mem_size]
  

  This reduces MPI's internal bookkeeping and simplifies synchronization.

• Lighter-Weight Synchronization Than Barriers: If your access pattern allows (e.g., producer-consumer or single-writer-multiple-reader), consider replacing global barriers with window-specific operations:

  • MPI_Win_sync: This operation flushes the local cache to ensure memory operations are visible to other processes (or vice versa). It's lighter than a full barrier, though you may still need a barrier if you require strict global ordering of read/write phases.
  • MPI_Win_fence: Instead of a global barrier, using MPI_Win_fence(0, win) on a single window can synchronize access to that window with lower overhead than a communicator-wide barrier.

• Leverage MPI Implementation-Specific Optimizations: Check your MPI library's documentation (e.g., Open MPI, MPICH) for MPI_INFO parameters that can optimize shared memory allocation. For example:

  • Enable huge pages for larger, more efficient memory access.
  • Set alignment parameters to match CPU cache lines and avoid false sharing.

内容的提问来源于stack exchange，提问作者nat chouf