从大文件生成小文件时遇System.OutOfMemoryException错误求解决

Hi there, let's figure out why you're hitting that System.OutOfMemoryException even though you've handled larger files before. The issue is likely not the total file size, but how PowerShell is parsing the content of this specific .tmp file. Here are actionable fixes to try:

• Use a .NET StreamReader/Writer for granular control
  This avoids PowerShell's default buffering behavior, which can choke on extra-long lines or non-standard file structures. The script reads one line at a time and writes it immediately, keeping memory usage minimal:

  $inputPath = "input_file_name.Tmp"
  $outputPath = "output_file_name_100.Tmp"
  $targetLines = 100
  
  # Initialize readers/writers with your specified default encoding
  $reader = [System.IO.StreamReader]::new($inputPath, [System.Text.Encoding]::Default)
  $writer = [System.IO.StreamWriter]::new($outputPath, $false, [System.Text.Encoding]::Default)
  
  try {
      for ($i = 0; $i -lt $targetLines; $i++) {
          $line = $reader.ReadLine()
          if ($null -eq $line) { break } # Exit early if we hit the end of the file
          $writer.WriteLine($line)
      }
  } finally {
      # Ensure we clean up resources even if an error occurs
      $reader.Close()
      $writer.Close()
  }
  

• Force line-by-line reading with Get-Content
  By default, Get-Content batches lines into memory for efficiency. Adding -ReadCount 1 forces it to process one line at a time, which can prevent memory spikes if your .tmp file has unusually long lines:

  Get-Content input_file_name.Tmp -ReadCount 1 -TotalCount 100 | Out-File -Encoding Default "output_file_name_100.Tmp"
  

• Verify the file's structure and encoding
  Your older files probably had regular line breaks, so -TotalCount 100 only loaded the first 100 distinct lines. This new .tmp file might be:

  • A binary file with no line breaks (so Get-Content tries to load the entire file as a single "line")
  • Using an encoding that Default doesn't parse correctly, leading to corrupted line detection
    You can check for line breaks with a quick command: Get-Content input_file_name.Tmp -TotalCount 1 | Measure-Object -Character — if the character count is massive, you're dealing with an ultra-long line or binary data.

• Upgrade to PowerShell 7+
  The newer cross-platform PowerShell has improved memory management for file operations. If you're still using Windows PowerShell 5.1, switching to PowerShell 7 might resolve the issue without changing your original command.

内容的提问来源于stack exchange，提问作者Chethan Simha