优化大体积DNS日志处理PowerShell脚本的运行速度

我有一个PowerShell脚本，功能正常、输出完全符合需求，但处理500MB左右、包含600万行的DNS日志时耗时过长。虽明白大数据处理需要时间，但仍希望找到优化方法。以下是精简后的脚本和DNS日志示例：

原脚本 #

$DnsFilePath = "C:\dns.log"

Param([string]$DnsFilePath)
If (Test-Path $DnsFilePath) 
    { 
        $FileInfo = Get-ChildItem -Path $DnsFilePath
        $Ans = Read-Host "Do you want to continue(y/n)?"
        
        If ($Ans -eq 'y')
            {
                If (!($SkipLines)) { Write-Host "Processing..."; }
                $i = 0; ## 记录处理的条目数
                $Timer= [Diagnostics.Stopwatch]::StartNew() ## 启动计时器
                $ArrayOfStrings = [System.Collections.ArrayList]@()

                Switch -regex ([System.IO.File]::ReadLines($FileInfo.fullname)) {
                ' UDP Rcv ' {
                    $Datetime = [regex]::matches($switch.current,'\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} (AM|PM)').Value
                    $IP = [regex]::matches($switch.current,'\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b').Value
                    $FQDN = [regex]::matches($switch.current,"\)[A-z0-9-_]*\(").Value  -replace "\)|\(","" -join "."
                    [void]$ArrayOfStrings.Add("$Datetime,$IP,$FQDN")
                    $i++;
                            }
                }
                        $OutFilePath = "$($FileInfo.DirectoryName)\$($FileInfo.BaseName)_Parsed.txt"
                        [System.IO.File]::WriteAllLines($OutFilePath, $ArrayOfStrings)
                        $Timer.stop()
                        Write-host "Total time elapsed: $($Timer.Elapsed.ToString('hh\:mm\:ss\.ff'))"
                        Write-Host "Number of Record Processed: $i"
                        Write-Host "Parsed File created successfully at $OutFilePath"                       
            }
        else
            { Write-Host "Script exits." }
    }
Else
    {
    Write-Host -fore Red "File does not exist in the following location: $DnsFilePath. Script exits."
    }

DNS日志示例 #

DNS Server log file creation at 7/10/2023 10:55:42 AM
Log file wrap at 7/10/2023 10:55:42 AM

Message logging key (for packets - other items use a subset of these fields):
    Field #  Information         Values
    -------  -----------         ------
       1     Date
       2     Time
       3     Thread ID
       4     Context
       5     Internal packet identifier
       6     UDP/TCP indicator
       7     Send/Receive indicator
       8     Remote IP
       9     Xid (hex)
      10     Query/Response      R = Response
                                 blank = Query
      11     Opcode              Q = Standard Query
                                 N = Notify
                                 U = Update
                                 ? = Unknown
      12     [ Flags (hex)
      13     Flags (char codes)  A = Authoritative Answer
                                 T = Truncated Response
                                 D = Recursion Desired
                                 R = Recursion Available
      14     ResponseCode ]
      15     Question Type
      16     Question Name

7/10/2023 10:55:42 AM 1B7C PACKET  000001D9D88C68D0 UDP Rcv 8.8.8.8         5fb1 R Q [8381   DR NXDOMAIN] A      (3)www(12)autodiscover(5)st1ad(4)emea(15)microsoftonline(3)com(0)

7/10/2023 10:55:42 AM 1B7C PACKET  000001D9D775F890 UDP Snd 10.x.x.x     92cb R Q [8381   DR NXDOMAIN] A      (3)www(12)autodiscover(5)st1ad(4)emea(15)microsoftonline(3)com(0)

7/10/2023 10:55:42 AM 1B7C PACKET  000001D9E4E338D0 UDP Rcv 10.x.x.x  a9bd   Q [0001   D   NOERROR] A      (18)addinsinstallation(5)store(6)office(3)com(0)

7/10/2023 10:55:42 AM 1B7C PACKET  000001D9D775F890 UDP Snd 8.8.8.8         afda   Q [0001   D   NOERROR] A      (23)prod-addinsinstallation(15)omexexternallfb(6)office(3)net(6)akadns(3)net(0)

7/10/2023 10:55:42 AM 1B78 PACKET  000001D9E182BB80 UDP Rcv 10.x.x.x  d229   Q [0001   D   NOERROR] SOA    (15)pc_host01(7)contoso(5)local(0)

7/10/2023 10:55:42 AM 1B78 PACKET  000001D9E182BB80 UDP Snd 10.x.x.x  d229 R Q [8085 A DR  NOERROR] SOA    (15)pc_host02(7)contoso(5)local(0)

7/10/2023 10:55:42 AM 1B78 PACKET  000001D9E2A2D670 UDP Rcv 8.8.8.8         c95c R Q [8081   DR  NOERROR] A      (9)dtr-a-ncu(2)na(8)azurerms(3)com(0)

7/10/2023 10:55:42 AM 1B78 PACKET  000001D9E1998D80 UDP Snd 10.x.x.x     2047 R Q [8081   DR  NOERROR] A      (6)portal(8)azurerms(3)com(0)

7/10/2023 10:55:42 AM 1B78 PACKET  000001D9E2D07D00 UDP Rcv 10.x.x.x   788e   Q [0001   D   NOERROR] A      (2)tr(11)c1182306347(12)ip4-58f0802d(4)wgcs(7)skyhigh(5)cloud(0)

7/10/2023 10:55:42 AM 1B78 PACKET  000001D9E1998D80 UDP Snd 8.8.8.8         1c22   Q [0001   D   NOERROR] A      (2)tr(11)c1182306347(12)ip4-58f0802d(4)wgcs(7)skyhigh(5)cloud(0)

优化方案 #

针对大文件场景，主要从内存占用和正则匹配效率两个核心方向优化：

1. 预编译正则表达式 #

避免每次匹配都重新生成正则对象，提前编译并复用，减少重复开销：

$regexDatetime = [regex]::new('\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} (AM|PM)', [System.Text.RegularExpressions.RegexOptions]::Compiled)
$regexIP = [regex]::new('\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b', [System.Text.RegularExpressions.RegexOptions]::Compiled)
$regexFQDNParts = [regex]::new('\((\d+)\)([A-z0-9-_]+)', [System.Text.RegularExpressions.RegexOptions]::Compiled)

2. 直接逐行写入输出文件 #

放弃用ArrayList缓存所有结果，改用文件流逐行写入，大幅降低内存占用，避免百万级数据的内存扩容开销：

$OutFilePath = "$($FileInfo.DirectoryName)\$($FileInfo.BaseName)_Parsed.txt"
$writer = [System.IO.StreamWriter]::new($OutFilePath)

3. 简化FQDN解析逻辑 #

用单个正则捕获所有FQDN片段，直接拼接成完整域名，避免多次替换和合并操作：

$fqdnParts = $regexFQDNParts.Matches($line)
$FQDN = ($fqdnParts | ForEach-Object { $_.Groups[2].Value }) -join '.'

优化后的完整脚本 #

Param([string]$DnsFilePath = "C:\dns.log")

If (Test-Path $DnsFilePath) 
{ 
    $FileInfo = Get-ChildItem -Path $DnsFilePath
    $Ans = Read-Host "Do you want to continue(y/n)?"
    
    If ($Ans -eq 'y')
    {
        If (!($SkipLines)) { Write-Host "Processing..."; }
        $recordCount = 0
        $timer = [Diagnostics.Stopwatch]::StartNew()

        # 预编译所有需要的正则表达式
        $regexFilter = [regex]::new(' UDP Rcv ', [System.Text.RegularExpressions.RegexOptions]::Compiled)
        $regexDatetime = [regex]::new('\d{1,2}/\d{1,2}/\d{4} \d{1,2}:\d{1,2}:\d{1,2} (AM|PM)', [System.Text.RegularExpressions.RegexOptions]::Compiled)
        $regexIP = [regex]::new('\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b', [System.Text.RegularExpressions.RegexOptions]::Compiled)
        $regexFQDNParts = [regex]::new('\((\d+)\)([A-z0-9-_]+)', [System.Text.RegularExpressions.RegexOptions]::Compiled)

        $OutFilePath = "$($FileInfo.DirectoryName)\$($FileInfo.BaseName)_Parsed.txt"
        # 初始化文件写入流
        $writer = [System.IO.StreamWriter]::new($OutFilePath)

        try {
            foreach ($line in [System.IO.File]::ReadLines($FileInfo.FullName)) {
                if ($regexFilter.IsMatch($line)) {
                    $datetime = $regexDatetime.Match($line).Value
                    $ip = $regexIP.Match($line).Value
                    $fqdnParts = $regexFQDNParts.Matches($line)
                    $fqdn = ($fqdnParts | ForEach-Object { $_.Groups[2].Value }) -join '.'
                    
                    $writer.WriteLine("$datetime,$ip,$fqdn")
                    $recordCount++
                }
            }
        }
        finally {
            $writer.Dispose() # 确保文件流被正确关闭
        }

        $timer.Stop()
        Write-Host "Total time elapsed: $($timer.Elapsed.ToString('hh\:mm\:ss\.ff'))"
        Write-Host "Number of Record Processed: $recordCount"
        Write-Host "Parsed File created successfully at $OutFilePath"                       
    }
    else
    { 
        Write-Host "Script exits." 
    }
}
Else
{
    Write-Host -ForegroundColor Red "File does not exist in the following location: $DnsFilePath. Script exits."
}

内容的提问来源于stack exchange，提问作者Arbelac