合并D:\TILT_Logs日志文件时部分文件遗漏的技术求助

Hey Garrett, let's dig into why some of your log data is going missing when merging files with that PowerShell script. Here are the most likely issues and actionable fixes:

1. Mismatched Encodings Corrupting Data #

The default behavior of Get-Content and Set-Content uses PowerShell's system default encoding (usually UTF-16LE on Windows). If your source log files use other encodings (like UTF-8, ASCII, or regional formats), this mismatch can silently drop characters or truncate content.

Fix: Force a consistent encoding (UTF-8 is a safe universal choice):

$combine = Get-ChildItem 'D:\TILT_Logs' -Recurse -Filter '*.log'
Get-Content $combine.FullName -Encoding UTF8 | Set-Content 'D:\TILT_Logs\Combined.log' -Encoding UTF8

If you're unsure of the source files' encoding, you can test individual files with Get-Content -Raw -Encoding Byte to inspect the header, or use [System.Text.Encoding]::DetectEncodingFromByteOrderMarks() for automatic detection in bulk workflows.

2. Unhandled File Read Errors #

Some log files might be locked by another process, or you might lack read permissions. By default, Get-Content skips these files without any warning—so you won't know data is missing.

Fix: Add error handling to track problematic files:

$combine = Get-ChildItem 'D:\TILT_Logs' -Recurse -Filter '*.log'
$outputPath = 'D:\TILT_Logs\Combined.log'

# Clear existing output file if needed
if (Test-Path $outputPath) { Clear-Content $outputPath }

foreach ($file in $combine) {
    try {
        Get-Content $file.FullName -Encoding UTF8 -ErrorAction Stop | Add-Content $outputPath -Encoding UTF8
        Write-Host "Merged successfully: $($file.FullName)"
    }
    catch {
        Write-Warning "Failed to read file: $($file.FullName) → Error: $_"
    }
}

Using Add-Content instead of piping directly to Set-Content also ensures each file's content is appended properly, rather than overwriting the output file mid-process.

3. Memory Issues with Large Files #

If you're working with very large log files, Get-Content loading the entire file into memory at once can cause overflow, leading to partial writes or skipped content.

Fix: Use stream-based reading/writing to reduce memory load:

$combine = Get-ChildItem 'D:\TILT_Logs' -Recurse -Filter '*.log'
$outputPath = 'D:\TILT_Logs\Combined.log'

# Initialize a stream writer for efficient writing
$writer = [System.IO.StreamWriter]::new($outputPath, $false, [System.Text.Encoding]::UTF8)

foreach ($file in $combine) {
    try {
        # Read the file line by line instead of all at once
        foreach ($line in [System.IO.File]::ReadLines($file.FullName, [System.Text.Encoding]::UTF8)) {
            $writer.WriteLine($line)
        }
        Write-Host "Merged successfully: $($file.FullName)"
    }
    catch {
        Write-Warning "Failed to read file: $($file.FullName) → Error: $_"
    }
}

# Clean up the stream writer
$writer.Close()
$writer.Dispose()

This method is far more efficient for large datasets and avoids memory-related truncation.

4. Target File Locked by Another Program #

If Combined.log is open in another app (like Notepad, a log viewer, or another script) while you run your merge, Set-Content might fail to write all content or silently drop data.

Fix: Add a check to ensure the target file is accessible before starting:

$outputPath = 'D:\TILT_Logs\Combined.log'
if (Test-Path $outputPath) {
    try {
        # Attempt exclusive access to check if the file is locked
        $fileStream = [System.IO.File]::Open($outputPath, [System.IO.FileMode]::Open, [System.IO.FileAccess]::Write, [System.IO.FileShare]::None)
        $fileStream.Close()
    }
    catch {
        Write-Error "Target file $outputPath is locked by another program. Please close it and try again."
        exit 1
    }
}

# Proceed with merge code here...

内容的提问来源于stack exchange，提问作者Garrett