使用OpenMP在C语言中实现嵌套循环并行编程问题咨询

Hey there! Let's work through this problem since the collapse directive isn't going to cut it here—your sequential file reading step between the loops is non-negotiable, and that's exactly what breaks the collapse requirement of tightly nested loops without intervening code. Here are two practical solutions tailored to your scenario:

1. Preload All File Lines & Parallelize the Outer Loop #

If your file isn't astronomically large, the simplest approach is to first read all lines sequentially into memory, then parallelize the outer loop so each thread handles a subset of lines (along with their associated inner loop processing). This preserves your required intermediate code while leveraging multi-threading.

Example Code #

#include <omp.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

// Helper function to read all lines from a file into a dynamic array
int read_all_lines(const char* filename, char*** out_lines) {
    FILE* fp = fopen(filename, "r");
    if (!fp) return 0;

    char** lines = NULL;
    int count = 0;
    char buffer[4096]; // Adjust based on your maximum line length

    while (fgets(buffer, sizeof(buffer), fp)) {
        // Allocate memory for the current line
        char* line = malloc(strlen(buffer) + 1);
        strcpy(line, buffer);
        // Expand the lines array
        lines = realloc(lines, (count + 1) * sizeof(char*));
        lines[count++] = line;
    }

    fclose(fp);
    *out_lines = lines;
    return count;
}

int main() {
    char** lines;
    int total_lines = read_all_lines("your_input_file.txt", &lines);
    if (total_lines == 0) {
        fprintf(stderr, "Failed to read file\n");
        return 1;
    }

    // Set thread count (note: 81 threads on 20 cores may cause overhead—test with 20/40 first!)
    #pragma omp parallel num_threads(81)
    {
        // Parallelize the outer loop over preloaded lines
        #pragma omp for schedule(static)
        for (int i = 0; i < total_lines; i++) {
            // Your required intermediate code: process the current line's data
            // (e.g., parse values from lines[i], set up variables for the inner loop)
            // ...

            // Inner loop: process the line's content
            for (int j = 0; j < YOUR_INNER_LOOP_SIZE; j++) {
                // Do your per-element processing here
                // ...
            }
        }
    }

    // Clean up memory
    for (int i = 0; i < total_lines; i++) {
        free(lines[i]);
    }
    free(lines);
    return 0;
}

Why This Works #

• The sequential file read happens first, so you avoid race conditions from parallel file access.
• Each thread gets its own independent line to process, so no synchronization is needed unless your inner loop uses shared state.
• Your intermediate code stays intact, running right before the inner loop for each line.

2. Use Task Parallelism for On-Demand Processing #

If your file is too large to load entirely into memory, use OpenMP tasks to offload line processing to threads as you read each line sequentially. This keeps memory usage low while still parallelizing the heavy inner loop work.

Example Code #

#include <omp.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main() {
    FILE* fp = fopen("your_input_file.txt", "r");
    if (!fp) {
        fprintf(stderr, "Failed to open file\n");
        return 1;
    }

    char buffer[4096]; // Adjust line length as needed

    // Initialize thread pool
    #pragma omp parallel num_threads(81)
    {
        // Only one thread (the "single" thread) handles file reading
        #pragma omp single
        {
            while (fgets(buffer, sizeof(buffer), fp)) {
                // Copy the line to dynamic memory to avoid overwriting before the task runs
                char* line_copy = malloc(strlen(buffer) + 1);
                strcpy(line_copy, buffer);

                // Create a task to process this line
                #pragma omp task firstprivate(line_copy)
                {
                    // Your required intermediate code for this line
                    // ...

                    // Inner loop processing
                    for (int j = 0; j < YOUR_INNER_LOOP_SIZE; j++) {
                        // Process elements from line_copy
                        // ...
                    }

                    free(line_copy); // Clean up the copied line
                }
            }
        }
    }

    fclose(fp);
    return 0;
}

Why This Works #

• The main thread reads lines sequentially (preserving your required order of input), while worker threads handle the CPU-heavy inner loop processing.
• Tasks are distributed to idle threads automatically, so you make efficient use of your 20-core server.
• No need to load the entire file into memory—great for large datasets.

Key Notes to Keep in Mind #

• Thread Count Overhead: 81 threads on a 20-core server means many threads will be context-switching. Test with num_threads(20) (one per core) or num_threads(40) (if your server supports hyper-threading) to see which gives better performance.
• Thread Safety: If your intermediate code or inner loop uses shared variables (like global counters), wrap those sections in #pragma omp critical or use atomic operations to avoid race conditions.
• Memory Management: In the task-based approach, always copy lines to dynamic memory—don't pass the stack-based buffer to tasks, since it will be overwritten by the next read before the task completes.

内容的提问来源于stack exchange，提问作者Sanjiv Pradhanang