MPI Scatter结合std::vector实现矩阵分组分发异常排查

编译运行命令：

mpic++ main.cpp && mpiexec -np 4 ./a.out

需求说明 #

需要根据进程rank数量动态划分矩阵，示例矩阵：

std::vector<std::vector<int>> A = {
      {0, 0, 0, 0}, {1, 1, 1, 1}, {2, 2, 2, 2}, {3, 3, 3, 3}};

当分组数为2时，期望结果：

• 进程0：本地矩阵A = ((0,0,0,0), (1,1,1,1))
• 进程1：本地矩阵A = ((0,0,0,0), (1,1,1,1))
• 进程2：本地矩阵A = ((2,2,2,2), (3,3,3,3))
• 进程3：本地矩阵A = ((2,2,2,2), (3,3,3,3))

原始代码 #

#include <mpi.h>

#include <iostream>
#include <vector>

void print_matrix(const std::vector<std::vector<int>>& mat,
                  const std::string& name, int rank) {
  std::cout << "Process " << rank << " received matrix " << name
            << " with dimensions: " << mat.size() << "x"
            << (mat.empty() ? 0 : mat[0].size()) << std::endl;
  std::cout << "Process " << rank << " printing " << name << ":" << std::endl;
  for (const auto& row : mat) {
    for (int val : row) {
      std::cout << val << " ";
    }
    std::cout << std::endl;
  }
}

int main(int argc, char** argv) {
  MPI_Init(&argc, &argv);
  int rank, size;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);

  int n = 4;                  // Assuming a 4x4 matrix
  int rows_per_proc = n / 2;  // Each process handles half the rows

  MPI_Comm group_comm;
  int group = rank % 2;
  MPI_Comm_split(MPI_COMM_WORLD, group, rank, &group_comm);

  int new_rank;
  MPI_Comm_rank(group_comm, &new_rank);

  std::vector<std::vector<int>> A = {
      {0, 0, 0, 0}, {1, 1, 1, 1}, {2, 2, 2, 2}, {3, 3, 3, 3}};
  std::vector<std::vector<int>> localA(rows_per_proc, std::vector<int>(n));

  // Create a datatype for a block of rows
  MPI_Datatype matrix_block;
  MPI_Type_vector(rows_per_proc, n, n, MPI_INT, &matrix_block);
  MPI_Type_commit(&matrix_block);

  int root = 0;

  // Use MPI_Scatter with the created datatype
  if (new_rank == root) {
    MPI_Scatter(&A[0][0], 1, matrix_block, &localA[0][0], 1, matrix_block, root,
                group_comm);
  } else {
    MPI_Scatter(nullptr, 0, matrix_block, &localA[0][0], 1, matrix_block, root,
                group_comm);
  }

  print_matrix(localA, "Matrix A", rank);

  // Free the custom datatype
  MPI_Type_free(&matrix_block);

  MPI_Finalize();
  return 0;
}

错误输出 #

实际运行输出：

Process 0 received matrix Matrix A with dimensions: 2x4
Process 0 printing Matrix A:
0 0 0 0 
0 0 0 0 
Process 1 received matrix Matrix A with dimensions: 2x4
Process 1 printing Matrix A:
0 0 0 0 
0 0 0 0 
Process 2 received matrix Matrix A with dimensions: 2x4
Process 2 printing Matrix A:
1 1 1 1 
0 0 0 0 
Process 3 received matrix Matrix A with dimensions: 2x4
Process 3 printing Matrix A:
1 1 1 1 
0 0 0 0 

问题分析与修正 #

原始代码存在两个核心问题：

1. 数据分发逻辑错误：使用MPI_Scatter会将根进程的数据拆分后分发给组内每个进程，但需求是把整组对应的行块广播给组内所有进程，而非拆分。
2. 组内数据源错误：组1（rank=2、3）的根进程未从矩阵的第2行开始取数据，导致分发内容错误。

修正后代码 #

#include <mpi.h>

#include <iostream>
#include <vector>

void print_matrix(const std::vector<std::vector<int>>& mat,
                  const std::string& name, int rank) {
  std::cout << "Process " << rank << " received matrix " << name
            << " with dimensions: " << mat.size() << "x"
            << (mat.empty() ? 0 : mat[0].size()) << std::endl;
  std::cout << "Process " << rank << " printing " << name << ":" << std::endl;
  for (const auto& row : mat) {
    for (int val : row) {
      std::cout << val << " ";
    }
    std::cout << std::endl;
  }
}

int main(int argc, char** argv) {
  MPI_Init(&argc, &argv);
  int rank, size;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);

  int n = 4;                  // 4x4矩阵
  int rows_per_group = n / 2; // 每个组处理的行数

  MPI_Comm group_comm;
  int group = rank % 2;
  MPI_Comm_split(MPI_COMM_WORLD, group, rank, &group_comm);

  int new_rank;
  MPI_Comm_rank(group_comm, &new_rank);

  std::vector<std::vector<int>> A;
  std::vector<std::vector<int>> localA(rows_per_group, std::vector<int>(n));

  // 仅组内根进程需要持有完整矩阵
  if (rank == 0 || rank == 2) {
    A = {
        {0, 0, 0, 0}, {1, 1, 1, 1}, {2, 2, 2, 2}, {3, 3, 3, 3}};
  }

  // 创建自定义数据类型：连续rows_per_group行，每行n个int
  MPI_Datatype matrix_block;
  MPI_Type_vector(rows_per_group, n, n, MPI_INT, &matrix_block);
  MPI_Type_commit(&matrix_block);

  int root = 0;
  // 每个组的根进程广播对应的数据块
  if (group == 0) {
    // 组0广播前2行
    MPI_Bcast(&A[0][0], 1, matrix_block, root, group_comm);
  } else {
    // 组1广播后2行，数据源从A[2][0]开始
    MPI_Bcast(&A[2][0], 1, matrix_block, root, group_comm);
  }

  print_matrix(localA, "Matrix A", rank);

  MPI_Type_free(&matrix_block);
  MPI_Comm_free(&group_comm); // 释放创建的通信子

  MPI_Finalize();
  return 0;
}

修正说明 #

• 替换MPI_Scatter为MPI_Bcast：广播操作能让组内所有进程接收相同的行块，符合需求。
• 为每个组指定正确数据源：组0从矩阵第0行开始广播，组1从第2行开始广播。
• 优化内存使用：仅组内根进程持有完整矩阵，其他进程无需存储冗余数据。
• 新增MPI_Comm_free：释放MPI_Comm_split创建的通信子，避免资源泄漏。

内容的提问来源于stack exchange，提问作者BestSithInEU