如何以最快速度并行填充多个List并将其加入队列？求最佳实践控制台应用示例

Great question! When you're trying to populate multiple lists as quickly as possible while maximizing CPU usage, parallel processing is your best bet—since generating random numbers is CPU-bound work, spreading this across multiple cores will cut down total time significantly. Let's break down the approach and walk through a concrete example.

To hit your goals (max CPU utilization, populate 10 lists of 5k random numbers each, enqueue each list once complete), you'll want to focus on these key points:

• Parallelize independent tasks: Each list's population is a self-contained job, perfect for distributing across CPU cores.
• Use thread-safe structures: The queue that receives completed lists needs to handle concurrent writes without corruption.
• Optimize per-task performance: Avoid unnecessary overhead like dynamic list resizing or thread contention for shared resources (like a single Random instance).
• Control concurrency: Match the number of parallel tasks to your CPU core count to avoid excessive context switching.

This example uses .NET's Task Parallel Library (TPL) to handle parallel execution, a ConcurrentQueue for thread-safe enqueuing, and optimizations to keep CPU utilization high:

using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;

class ParallelListPopulator
{
    // Thread-safe queue to hold completed lists
    private static readonly ConcurrentQueue<List<int>> _completedListsQueue = new ConcurrentQueue<List<int>>();

    static void Main(string[] args)
    {
        const int totalLists = 10;
        const int itemsPerList = 5000;

        Console.WriteLine($"Starting parallel population of {totalLists} lists (each with {itemsPerList} random numbers)...\n");

        // Use Parallel.For to distribute work across CPU cores
        Parallel.For(0, totalLists, new ParallelOptions 
        { 
            // Limit parallelism to CPU core count for optimal utilization
            MaxDegreeOfParallelism = Environment.ProcessorCount 
        }, listIndex =>
        {
            // Pre-allocate list capacity to avoid dynamic resizing overhead
            var currentList = new List<int>(itemsPerList);
            // Create a thread-local Random instance to avoid contention
            var localRandom = GetThreadSafeRandom();

            // Populate the list
            for (int i = 0; i < itemsPerList; i++)
            {
                currentList.Add(localRandom.Next(0, 100000));
            }

            // Enqueue the completed list (thread-safe operation)
            _completedListsQueue.Enqueue(currentList);
            Console.WriteLine($"List {listIndex + 1} populated and added to queue.");
        });

        // Process the queue to demonstrate it works
        Console.WriteLine("\nProcessing completed lists from queue:");
        int processedCount = 0;
        while (_completedListsQueue.TryDequeue(out var list))
        {
            processedCount++;
            Console.WriteLine($"Processed List {processedCount}: {list.Count} items | Min: {list.Min()} | Max: {list.Max()}");
        }

        Console.WriteLine("\nAll tasks completed successfully!");
        Console.ReadKey();
    }

    // Helper to create a thread-safe Random instance (avoids shared state contention)
    private static Random GetThreadSafeRandom()
    {
        // Use a unique seed per instance to prevent duplicate number sequences
        int seed = Guid.NewGuid().GetHashCode();
        return new Random(seed);
    }
}

Let's unpack why each part of this example matters:

• Parallel.For with MaxDegreeOfParallelism: This automatically manages task scheduling across your CPU cores. Setting the limit to Environment.ProcessorCount ensures you don't create more threads than your CPU can handle, which would waste cycles on context switching.
• ConcurrentQueue: Unlike a regular Queue<T>, this collection is designed for concurrent read/write operations—no need to add manual locks, which would introduce bottlenecks.
• Pre-allocated List Capacity: Initializing List<int> with itemsPerList tells the runtime to allocate enough memory upfront, avoiding expensive resizing operations as the list grows.
• Thread-local Random: The standard Random class isn't thread-safe. If multiple tasks share one instance, you'll get duplicate numbers and performance hits from lock contention. Creating a unique Random per task solves this.

• Error Handling: If your list population logic could throw exceptions, wrap the work in a try/catch block inside the Parallel.For delegate. You can also check the ParallelLoopResult returned by Parallel.For to see if any tasks failed.
• Adjust Concurrency: If you're running on a system with hyper-threading, you might experiment with setting MaxDegreeOfParallelism to Environment.ProcessorCount * 2—but test first, as hyper-threading doesn't always double performance for CPU-bound work.
• For Other Languages: If you're using Python, you'd use the multiprocessing module (instead of threads, due to the GIL) and a Queue from the same module. For Java, ExecutorService and ConcurrentLinkedQueue would be your tools.

内容的提问来源于stack exchange，提问作者Larry