基于TPL并行数据流实现MD5文件哈希进度条上报方案咨询

问题背景 #

基于TPL并行数据流实现批量文件MD5哈希计算时，尝试将进度上报至UI的progressBar1控件，但进度值在0到文件总数间波动，无法得到0-100的准确百分比进度。运行环境为.NET Framework 4.8。

问题根源 #

1. 原代码中progressBar变量始终未被正确赋值，一直为0
2. 未提前获取总文件数，无法基于已处理文件占比计算进度
3. 进度上报逻辑依赖逐步增加的totalFilesFound，导致进度计算基准不断变化

修正后的实现代码 #

核心修改点 #

• 提前枚举所有目标文件，获取总文件数作为进度计算基准
• 基于已完成哈希的文件数与总文件数的比例，计算0-100的百分比进度
• 保留原有的限流上报逻辑，避免UI频繁刷新

public static class Example
{
    private class Dto
    {
        public Dto(string filePath, byte[] data)
        {
            FilePath = filePath;
            Data = data;
        }

        public string FilePath { get; }
        public byte[] Data { get; }
    }

    public static async Task ProcessFiles(string path, IProgress<ProgressReport> progress)
    {
        // 提前获取所有要处理的文件，得到总文件数
        var allFiles = Directory.EnumerateFiles(path).ToList();
        int totalFilesTotal = allFiles.Count;
        
        int totalFilesRead = 0;
        int totalFilesHashed = 0;
        int totalFilesUploaded = 0;

        DateTime lastReported = DateTime.UtcNow;

        void ReportProgress()
        {
            if (DateTime.UtcNow - lastReported < TimeSpan.FromSeconds(1))
            {
                return;
            }

            lastReported = DateTime.UtcNow;
            
            // 计算0-100的百分比进度，避免除零错误
            int progressValue = totalFilesTotal == 0 
                ? 0 
                : (int)((double)totalFilesHashed / totalFilesTotal * 100);

            var report = new ProgressReport(
                totalFilesTotal, 
                totalFilesRead, 
                totalFilesHashed, 
                totalFilesUploaded, 
                progressValue);
            progress.Report(report);
        }

        var getFilesBlock = new TransformBlock<string, Dto>(filePath =>
        {
            var dto = new Dto(filePath, File.ReadAllBytes(filePath));
            Interlocked.Increment(ref totalFilesRead); // 统一用Interlocked保证线程安全
            return dto;
        });

        var hashFilesBlock = new TransformBlock<Dto, Dto>(inDto =>
        {
            using var md5 = MD5.Create();
            var outDto = new Dto(inDto.FilePath, md5.ComputeHash(inDto.Data));
            Interlocked.Increment(ref totalFilesHashed);
            ReportProgress();
            return outDto;
        }, new ExecutionDataflowBlockOptions
        {
            MaxDegreeOfParallelism = Environment.ProcessorCount,
            BoundedCapacity = 50
        });

        var writeToDatabaseBlock = new ActionBlock<Dto>(arg =>
        {
            // 写入数据库逻辑
            Interlocked.Increment(ref totalFilesUploaded);
            ReportProgress();
        }, new ExecutionDataflowBlockOptions
        {
            BoundedCapacity = 50
        });

        getFilesBlock.LinkTo(hashFilesBlock, new DataflowLinkOptions { PropagateCompletion = true });
        hashFilesBlock.LinkTo(writeToDatabaseBlock, new DataflowLinkOptions { PropagateCompletion = true });

        // 发送所有文件到处理块
        foreach (var filePath in allFiles)
        {
            await getFilesBlock.SendAsync(filePath).ConfigureAwait(false);
        }

        getFilesBlock.Complete();
        await writeToDatabaseBlock.Completion.ConfigureAwait(false);
    }
}

public class ProgressReport
{
    public ProgressReport(int totalFilesTotal, int totalFilesRead, int totalFilesHashed, int totalFilesUploaded, int progress)
    {
        TotalFilesTotal = totalFilesTotal;
        TotalFilesRead = totalFilesRead;
        TotalFilesHashed = totalFilesHashed;
        TotalFilesUploaded = totalFilesUploaded;
        Progress = progress;
    }

    public int TotalFilesTotal { get; }
    public int TotalFilesRead { get; }
    public int TotalFilesHashed { get; }
    public int TotalFilesUploaded { get; }
    public int Progress { get; }
}

UI调用代码 #

private async void button1_Click(object sender, EventArgs e)
{
    IProgress<ProgressReport> progress = new Progress<ProgressReport>(value => 
    {
        progressBar1.Value = value.Progress;
        // 可选：显示详细统计信息
        // labelStatus.Text = $"已哈希：{value.TotalFilesHashed}/{value.TotalFilesTotal}";
    });

    await Example.ProcessFiles(@".\Downloads", progress);
    MessageBox.Show("文件哈希处理完成");
}

额外说明 #

• 如果目标目录文件数量极大，提前枚举所有文件可能占用较多内存，可以改为先遍历一次统计总数，再遍历一次处理文件，避免一次性加载所有文件路径到内存
• 若需要基于文件大小计算进度（而非文件数量），可以提前统计所有文件的总字节数，在读取文件时累加已读取字节数，再计算占比

内容的提问来源于stack exchange，提问作者WLFree