VB.NET并行下载多URL内容全部存入同一文件的问题如何解决?
问题原因
- 你把
name、t、local、orfWriter、s、r这些需要每个下载任务独立使用的变量声明成了类级别全局变量,并行执行时多个线程会同时修改这些变量的取值,后启动的线程会覆盖前一个线程写入的name值,最终所有下载任务写文件时用的Name都是最后一个任务设置的值,自然所有数据都写到同一个文件里。 DownloadAsync是返回Task的异步方法,但你在Parallel.ForEach的委托里直接调用没有等待执行完成,会出现Parallel.ForEach循环已经跑完、提示“All Done”的时候,实际下载任务还没全部执行完毕的问题。而且你调用异步方法用了.Result会阻塞线程,相当于白写了异步逻辑。- 额外小问题:代码里没有显式声明
name变量,严格模式下会编译报错,而且文件流操作没有用Using包裹,异常时可能出现资源泄露。
修复方案
你需要把所有任务独立的变量改成局部变量,调整异步调用逻辑,修改后的完整代码如下:
Imports System.Net Public Class Form1 Private Async Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click Dim fileReader As String = My.Computer.FileSystem.ReadAllText("G:\test.txt") RichTextBox1.Text = fileReader Dim urls As String() = RichTextBox1.Lines.Where(Function(line) Not String.IsNullOrEmpty(line)).ToArray() ' 改用Task.WhenAll等待所有异步下载完成 Dim downloadTasks As New List(Of Task)() For Each line In urls Dim parts = line.Split(",") Dim fileName = parts(0).Trim() Dim url = parts(1).Trim() downloadTasks.Add(DownloadAsync(url, fileName)) Next Await Task.WhenAll(downloadTasks) My.Computer.Audio.PlaySystemSound(System.Media.SystemSounds.Asterisk) MsgBox("All Done!") End Sub Async Function DownloadAsync(URL As String, fileName As String) As Task(Of Boolean) ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 Try Dim request As HttpWebRequest = HttpWebRequest.Create(URL) request.AutomaticDecompression = DecompressionMethods.GZip request.Timeout = 500 request.Method = "GET" request.UserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X x.y; rv:42.0) Gecko/20100101 Firefox/42.0" Using response As WebResponse = Await request.GetResponseAsync() Using ioStream As IO.Stream = response.GetResponseStream() Using sr As New System.IO.StreamReader(ioStream) Dim html As String = Await sr.ReadToEndAsync() Dim s As String() = html.Split(";") Dim r As String() = s(0).Split(",") Dim localPath As String = $"G:\{fileName}.csv" ' 用Using包裹文件流自动释放资源 Using orfWriter As New System.IO.StreamWriter(localPath, True, System.Text.Encoding.Unicode) Dim strami = $"{r(0)},{r(1)},{r(2)},{r(3)}" Await orfWriter.WriteLineAsync(strami) End Using End Using End Using End Using Return True Catch ex As Exception ' 可自行添加下载失败的日志记录逻辑 Return False End Try End Function Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load End Sub End Class
核心修改点说明
- 删掉了所有类级别的共享变量,把每个下载任务需要用到的
fileName、url、html、文件路径等全部改成方法内局部变量,每个线程的变量独立不会互相覆盖 - 把文件名
fileName作为参数直接传入DownloadAsync方法,不需要依赖全局变量传值,彻底避免变量覆盖问题 - 把
Parallel.ForEach改成普通循环加Task.WhenAll,适配异步方法的执行逻辑,避免线程浪费,同时保证所有下载任务执行完成后才弹出完成提示 - 所有异步操作统一用
Await关键字等待,去掉阻塞线程的.Result调用,提升运行效率 - 所有IO操作(网络流、文件流)全部用
Using包裹,出现异常时会自动释放资源,不会出现文件被锁定的问题
内容的提问来源于stack exchange,提问作者AGH
相关产品推荐
相关产品推荐

