使用System.IO.Compression.GzipStream并行处理20GB+超大文件压缩解压的技术咨询

Hey there! Let's tackle your problem step by step—handling 20GB+ files with .NET Framework's GzipStream while keeping memory usage in check and adding parallel processing. Since you're new to compression/decompression, I'll focus on practical, actionable code and explanations.

First things first: we can't load the entire file into memory, so streaming is non-negotiable. GzipStream is designed exactly for this—it works directly with input/output streams, processing data in chunks instead of loading everything at once.

Here's a minimal example of compressing a large file without loading it into memory. We'll use a buffer to read chunks from the source file, pass them through GzipStream, and write to the compressed file:

using System.IO;
using System.IO.Compression;
using System.Threading.Tasks;

public static async Task CompressLargeFile(string sourcePath, string destinationPath)
{
    // 64KB is a balanced buffer size—adjust based on your storage speed (e.g., 128KB for NVMe)
    const int bufferSize = 65536;

    using (var sourceStream = new FileStream(sourcePath, FileMode.Open, FileAccess.Read, FileShare.Read, bufferSize, useAsync: true))
    using (var destinationStream = new FileStream(destinationPath, FileMode.Create, FileAccess.Write, FileShare.None, bufferSize, useAsync: true))
    using (var gzipStream = new GZipStream(destinationStream, CompressionLevel.Optimal, leaveOpen: false))
    {
        // Copy data in chunks to avoid loading the whole file into memory
        await sourceStream.CopyToAsync(gzipStream, bufferSize);
    }
}

• The bufferSize controls how much data we read/write at a time—strike a balance between memory usage and I/O efficiency.
• Using useAsync: true leverages asynchronous I/O, keeping your app responsive without blocking threads.

Decompression follows the same streaming pattern—no full-file memory loads required:

public static async Task DecompressLargeFile(string sourcePath, string destinationPath)
{
    const int bufferSize = 65536;

    using (var sourceStream = new FileStream(sourcePath, FileMode.Open, FileAccess.Read, FileShare.Read, bufferSize, useAsync: true))
    using (var gzipStream = new GZipStream(sourceStream, CompressionMode.Decompress, leaveOpen: false))
    using (var destinationStream = new FileStream(destinationPath, FileMode.Create, FileAccess.Write, FileShare.None, bufferSize, useAsync: true))
    {
        await gzipStream.CopyToAsync(destinationStream, bufferSize);
    }
}

Important clarification: You can't parallelize compression/decompression of a single Gzip file—Gzip is a sequential format, where each chunk depends on the previous one. However, you can safely parallelize processing across multiple large files.

Here's how to process multiple files in parallel using async/await and Task.WhenAll:

using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;

public static async Task ParallelCompressFilesAsync(IEnumerable<string> sourceFilePaths, string outputDirectory)
{
    // Ensure the output directory exists before starting
    Directory.CreateDirectory(outputDirectory);

    // Create a task for each file compression
    var compressionTasks = sourceFilePaths.Select(filePath =>
    {
        var fileName = Path.GetFileName(filePath);
        var destinationPath = Path.Combine(outputDirectory, $"{fileName}.gz");
        return CompressLargeFile(filePath, destinationPath);
    });

    // Wait for all tasks to complete in parallel
    await Task.WhenAll(compressionTasks);
}

• To avoid overwhelming your system's CPU/I/O, limit parallelism with ParallelOptions if needed:

  // Use up to the number of CPU cores (adjust based on your hardware)
  var options = new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount };
  Parallel.ForEach(sourceFilePaths, options, filePath =>
  {
      var fileName = Path.GetFileName(filePath);
      var destinationPath = Path.Combine(outputDirectory, $"{fileName}.gz");
      CompressLargeFile(filePath, destinationPath).Wait();
  });
  

• Tune buffer size: For fast storage (like NVMe), increase the buffer to 128KB or 256KB to reduce I/O overhead.
• Pick the right compression level: Use CompressionLevel.Fastest if speed matters more than file size, or CompressionLevel.Optimal for a balanced approach.
• Reuse resources: If processing dozens of files, consider reusing buffers instead of creating new ones each time (though CopyToAsync handles this efficiently by default).
• Monitor resource usage: Use Visual Studio's Diagnostic Tools to ensure you're not accidentally loading large data chunks into memory.

• Forgetting to dispose streams: Always wrap streams in using statements—this ensures they're closed/flush properly, even if an error occurs.
• Trying to parallelize single-file compression: This will result in corrupted files or runtime errors, as Gzip relies on sequential data processing.
• Using tiny buffers: A 4KB buffer will trigger thousands of I/O operations, slowing down your code significantly.
• Ignoring async/await: Synchronous methods like CopyTo will block your main thread, making your app unresponsive during long operations.

Start with the single-file compression/decompression example first to get comfortable with streaming. Once you've verified that works, add parallel processing for multiple files. Don't hesitate to experiment with buffer sizes and compression levels to find what works best for your hardware and use case.

内容的提问来源于stack exchange，提问作者Radoslaw Jurewicz