基于MPI实现同步式TXT文件字符计数的技术咨询

Hey there! I see you've already nailed the file open/closed part—great start. Let's build on that to add the character counting functionality using MPI's file operations. Here's a step-by-step breakdown and updated code tailored to your task:

Key Steps to Follow #

• Get total file size: First, we need to know how big the file is to split the workload evenly across all processes.
• Divide the file into chunks: Calculate how many bytes each process should handle. The last process will pick up any remaining bytes if the file size isn't perfectly divisible by the number of processes.
• Jump to the process's starting position: Each process needs to navigate to its assigned section of the file.
• Read the chunk and count characters: For single-byte encodings like ASCII, each byte equals one character—so we can count the bytes directly (we’ll also cover how to adjust for excluding specific chars if needed).
• Aggregate results: Sum up the counts from all processes to get the total character count.

Updated Code with Explanations #

#include "stdafx.h"
#include "mpi.h"
#include <iostream>
using namespace std;

int _tmain(int argc, char *argv[]) {
    int rank, size;
    MPI_Status status;
    MPI_File fh;
    MPI_Offset file_size, offset, chunk_size;
    int local_count = 0;
    int total_count = 0;
    char *buffer;

    MPI_Init(&argc, &argv);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    MPI_Comm_size(MPI_COMM_WORLD, &size);

    // Open the file collectively (syncs all processes)
    MPI_File_open(MPI_COMM_WORLD, "file.txt", MPI_MODE_RDONLY, MPI_INFO_NULL, &fh);

    // Fetch total size of the file
    MPI_File_get_size(fh, &file_size);

    // Calculate base chunk size per process
    chunk_size = file_size / size;
    offset = rank * chunk_size;

    // Last process handles any leftover bytes
    if (rank == size - 1) {
        chunk_size = file_size - offset;
    }

    // Allocate buffer to hold the process's chunk
    buffer = new char[chunk_size];

    // Move to the starting position of this process's chunk
    MPI_File_seek(fh, offset, MPI_SEEK_SET);

    // Read the chunk into the buffer
    MPI_File_read(fh, buffer, chunk_size, MPI_CHAR, &status);

    // Count characters (direct byte count for single-byte encodings)
    local_count = chunk_size;
    // If you need to exclude specific chars (e.g., newlines), use this loop instead:
    // for (int i = 0; i < chunk_size; i++) {
    //     if (buffer[i] != '\n') {
    //         local_count++;
    //     }
    // }

    // Sum all local counts to get the total (MPI_Reduce is efficient for this)
    MPI_Reduce(&local_count, &total_count, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD);

    // Only rank 0 prints the final result
    if (rank == 0) {
        cout << "Total number of characters in the file: " << total_count << endl;
    }

    // Clean up allocated memory and close resources
    delete[] buffer;
    MPI_File_close(&fh);
    MPI_Finalize();
    return 0;
}

Important Notes #

• Collective Operations: We use collective MPI file calls (MPI_File_open, MPI_File_get_size) because they keep all processes synchronized, which is more efficient for distributed tasks.
• Character Encoding: This code works for single-byte encodings (ASCII, UTF-8 for basic characters). If you’re dealing with multi-byte characters (e.g., non-ASCII UTF-8), you’ll need to adjust the counting logic to parse multi-byte sequences instead of just counting bytes.
• Error Handling: For production code, add checks for MPI function return codes to catch issues like missing files or read errors. Example:

  int err = MPI_File_open(MPI_COMM_WORLD, "file.txt", MPI_MODE_RDONLY, MPI_INFO_NULL, &fh);
  if (err != MPI_SUCCESS) {
      if (rank == 0) cout << "Failed to open file!" << endl;
      MPI_Abort(MPI_COMM_WORLD, err);
  }
  

内容的提问来源于stack exchange，提问作者RaSiN