VB.NET中Microsoft.VisualBasic.FileIO.TextFieldParser大文件处理问题咨询

1. 用StreamReader逐行手动解析 #

TextFieldParser大概率是因为默认加载方式对大文件不友好，换成StreamReader逐行读取，自己处理字段分割（包括带引号的复杂字段），内存占用极低，完全适配大文件场景。

示例代码：

Using reader As New StreamReader("large_file.csv")
    Dim line As String
    While reader.Peek() >= 0
        line = reader.ReadLine()
        ' 调用辅助方法拆分带引号的字段
        Dim fields As String() = SplitQuotedFields(line, ",")
        ' 这里写你的业务处理逻辑
    End While
End Using

' 处理带引号的分隔字段，避免把引号内的分隔符当成字段分割符
Private Function SplitQuotedFields(line As String, delimiter As String) As String()
    Dim fields As New List(Of String)()
    Dim inQuote As Boolean = False
    Dim currentField As New StringBuilder()

    For Each c As Char In line
        If c = """"c Then
            inQuote = Not inQuote
        ElseIf c = delimiter(0) AndAlso Not inQuote Then
            fields.Add(currentField.ToString())
            currentField.Clear()
        Else
            currentField.Append(c)
        End If
    Next

    fields.Add(currentField.ToString())
    Return fields.ToArray()
End Function

2. 调整TextFieldParser的配置参数 #

有时候不是文件大小的锅，而是你没正确配置解析规则或者错误处理。试试以下设置，很多时候能解决大文件解析失败的问题：

Using parser As New TextFieldParser("large_file.csv")
    parser.TextFieldType = FieldType.Delimited
    parser.SetDelimiters(",") ' 根据你的文件分隔符调整
    parser.HasFieldsEnclosedInQuotes = True ' 如果字段用引号包裹，必须开这个
    ' 跳过无效行，避免因为某一行格式错误导致整个解析中断
    parser.ErrorAction = ErrorAction.SkipLine

    Dim currentRow As String()
    While Not parser.EndOfData
        Try
            currentRow = parser.ReadFields()
            ' 处理行数据
        Catch ex As MalformedLineException
            ' 可选：记录错误行信息用于排查
            Debug.WriteLine($"无效行 {parser.ErrorLineIndex}: {parser.ErrorLine}")
        End Try
    End While
End Using

3. 用第三方专业CSV解析库 #

如果上面两种方法都满足不了需求，直接上专门的CSV处理库，比如CsvHelper，它对大文件流式处理支持更好，能处理各种复杂CSV格式（比如字段内换行、嵌套引号等），比TextFieldParser健壮得多。

示例代码：

Imports CsvHelper
Imports System.Globalization

Using reader As New StreamReader("large_file.csv")
    Using csv As New CsvReader(reader, CultureInfo.InvariantCulture)
        ' 逐行读取解析
        While csv.Read()
            Dim field1 = csv.GetField(Of String)(0)
            Dim field2 = csv.GetField(Of Integer)(1)
            ' 业务处理逻辑
        End While

        ' 如果有实体类，还可以直接映射成对象列表，更方便
        ' csv.Configuration.RegisterClassMap(Of YourClassMap)()
        ' Dim records = csv.GetRecords(Of YourClass)()
    End Using
End Using

内容的提问来源于stack exchange，提问作者Nbl123