如何高效将大体积文本转存至数组？6GB分号分隔文本处理

Let’s work through how to efficiently process your 6GB semicolon-delimited .txt file, match against your pre-built dictionary, and store matching rows in a 2D array. The biggest priority here is avoiding loading the entire file into memory (which would crash most systems) and handling the array dynamically since we don’t know how many matches we’ll get upfront.

Key Ground Rules for Large Files #

• Never read the whole file at once—stick to line-by-line reading to keep memory usage low.
• VBScript arrays are fixed-size by default, so we’ll use ReDim Preserve to expand our 2D array as we find matches.
• Your dictionary lookup is already efficient (dictionaries have O(1) lookups), so we just need to pair that with smart file handling.

Revised, Efficient Code Implementation #

' Grab your pre-built dictionary (assuming dict_HB returns a valid Scripting.Dictionary)
Set hbDict = dict_HB(hb)

Set FSO = CreateObject("Scripting.FileSystemObject")
Const ForReading = 1

' Initialize our 2D array—start with 0 rows, we'll expand as needed
Dim matchArray()
ReDim matchArray(0, 0)
Dim currentMatchCount : currentMatchCount = 0

' Open the large file for line-by-line reading (critical for memory)
Set fileStream = FSO.OpenTextFile("C:\Your\File\Path\LargeFile.txt", ForReading, False)

Do Until fileStream.AtEndOfStream
    ' Read one line at a time—no massive memory hit here
    Dim line : line = fileStream.ReadLine
    
    ' Split the line into individual fields using semicolon as the delimiter
    Dim fields : fields = Split(line, ";")
    
    ' Replace INDEX_OF_TARGET_FIELD with the 0-based index of the field you want to check
    Dim targetField : targetField = fields(INDEX_OF_TARGET_FIELD)
    
    ' Check if the target field exists in our dictionary
    If hbDict.Exists(targetField) Then
        ' Expand the array to hold this new matching row
        currentMatchCount = currentMatchCount + 1
        ReDim Preserve matchArray(currentMatchCount - 1, UBound(fields))
        
        ' Copy all fields from the line into the array row
        Dim i
        For i = 0 To UBound(fields)
            matchArray(currentMatchCount - 1, i) = fields(i)
        Next
    End If
Loop

' Clean up resources to free memory
fileStream.Close
Set fileStream = Nothing
Set FSO = Nothing
Set hbDict = Nothing

' Example of how to iterate through the final 2D array
' Dim row, col
' For row = 0 To UBound(matchArray, 1)
'     For col = 0 To UBound(matchArray, 2)
'         WScript.Echo matchArray(row, col)
'     Next
' Next

Critical Notes to Adapt This to Your Use Case #

• Replace INDEX_OF_TARGET_FIELD: If your target field is the 3rd one (counting from 1), use 2 since VBScript uses 0-based indexing.
• Encoding Handling: If your file uses UTF-8 or non-ANSI encoding, add the encoding parameter to OpenTextFile—e.g., FSO.OpenTextFile("path", ForReading, False, 6) for UTF-8.
• Malformed Lines: If your file might have lines with missing fields, add On Error Resume Next (with proper error checking) to avoid crashes when accessing array indices.
• Extreme Match Volumes: If you expect millions of matches, consider writing matches to a temporary file instead of holding all in memory—this prevents hitting VBScript’s memory limits.

内容的提问来源于stack exchange，提问作者Musicodelic