使用Rayon并行处理CSV转JSON任务未达预期性能的问题排查与优化尝试

Hey there! Let's break down what's going on with your Rayon usage and why it wasn't matching std::thread's performance initially, plus some thoughts on your later results.

First, the big mistake in your initial Rayon code #

Your first Rayon implementation had a critical issue: pool.install() is a blocking call. It waits for the passed closure to finish executing before moving to the next iteration of the loop. That means your code was actually processing files serially, not in parallel!

No wonder the CPU utilization was only 98% (almost single-threaded) and total time was way longer than the std::thread version. You weren't using Rayon's parallelism at all here.

Why pool.scope() was better but still lagged #

Your switch to pool.scope() was the right move—scope.spawn() submits tasks to the thread pool without blocking the loop, so files are processed in parallel. But you noticed it still underperformed std::thread with 113 files. Here’s why that might happen:

• Rayon’s default thread count matches your CPU core count (4 for your quad-core i5). For IO-heavy tasks like reading/writing files, threads spend most of their time waiting on disk IO. Fewer threads mean CPU might sit idle while waiting for IO operations to finish.
• std::thread creates a new thread per file. While thread creation has overhead, for IO-bound work, the OS can schedule other threads to run while some are blocked on IO, keeping CPU busy. If your Rayon pool only had 4 threads, it couldn’t take advantage of that idle waiting time as effectively.

Your latest par_iter() implementation: close, but not identical #

Your updated into_par_iter().for_each() is Rayon’s standard, correct way to parallelize iterable tasks. Now performance is almost on par with std::thread—here’s why the small gap exists:

• Rayon adds minor overhead for task scheduling, load balancing, and thread pool management. This is negligible for most cases, but shows up in side-by-side comparisons.
• std::thread binds one file per thread, so there’s no task switching overhead per file. Thread creation/destruction cost is minimal here because IO time dominates total execution time.

Tips to narrow (or reverse) the gap #

1. Tune Rayon’s thread pool size for IO-bound work
   For tasks where threads wait on IO, set the thread count to 2-4x your CPU core count (e.g., 8-16 for your 4-core i5). This lets the CPU work on other tasks while some threads are blocked on disk. Example:

   rayon::ThreadPoolBuilder::new().num_threads(8).build_global()?;
   

2. Optimize the convert function itself
   The biggest gains will come from making CSV reading and JSON writing faster:

   • Use the csv crate’s optimized reader settings (e.g., ReaderBuilder::flexible(true) if your CSVs have minor formatting inconsistencies).
   • Write JSON directly to the file with serde_json::to_writer instead of building a large JSON string in memory first.

3. Test with larger file counts
   Rayon’s thread pool reuses threads, so as the number of files grows, the overhead of creating/destroying std::threads will add up. For hundreds of files, Rayon might start outperforming std::thread because it avoids that repeated thread setup cost.

内容的提问来源于stack exchange，提问作者fade2black