PowerShell远程实时日志监控重复告警问题求助

Hey there! Let's tackle this problem step by step—you're on the right track thinking about leveraging timestamps to cut down on those annoying repeat alerts. Here's a practical, scalable solution tailored to your scenario:

Core Idea #

Instead of scanning the entire log file every time, we'll only check for "Invalid Data" entries that occurred after our last check. We'll store a "last check timestamp" for each remote device, so we can filter out old errors and only alert on new ones.

Step-by-Step PowerShell Script #

This script handles remote log checking, timestamp filtering, alerting, and maintains state to prevent duplicates. I've added detailed comments since you're new to PowerShell:

# --------------------------
# Configuration Parameters
# --------------------------
$remoteDevices = @("Device01", "Device02", "Device03") # Replace with your full device list
$remoteLogPath = "\\{0}\C$\Path\To\Your\LogFile.log" # Remote log path template (use {0} for device name)
$smtpServer = "your-smtp-server.example.com"
$alertFrom = "system-alerts@yourdomain.com"
$alertTo = "it-admin@yourdomain.com"
$alertSubject = "CRITICAL: Invalid Data Detected on {0}"
$alertBodyTemplate = "New Invalid Data errors found on {0} (since {1}):`n`n{2}"

# Directory to store last check timestamps per device
$lastCheckDir = "C:\PowerShell-Alerts\LastChecks"
if (-not (Test-Path $lastCheckDir)) {
    New-Item -ItemType Directory -Path $lastCheckDir | Out-Null
}

# --------------------------
# Process Each Remote Device
# --------------------------
foreach ($device in $remoteDevices) {
    Write-Host "Checking logs for $device..."
    $targetLogPath = $remoteLogPath -f $device
    $lastCheckFile = Join-Path $lastCheckDir "$device.txt"

    # Get last check time (default to today's midnight if first run)
    if (Test-Path $lastCheckFile) {
        $lastCheckTime = Get-Content $lastCheckFile | Get-Date -ErrorAction Stop
    } else {
        $lastCheckTime = (Get-Date).Date # Start from today's 00:00 to skip historical logs
    }

    try {
        # Fetch log entries containing "Invalid Data"
        $errorEntries = Get-Content $targetLogPath -ErrorAction Stop | Where-Object { $_ -match "Invalid Data" }

        # Filter entries to only those after last check time
        $newErrors = @()
        foreach ($entry in $errorEntries) {
            # Match timestamp format: dd-MM-yyyy HH:mm:ss:fff
            if ($entry -match '^(\d{2}-\d{2}-\d{4} \d{2}:\d{2}:\d{2}:\d{3})') {
                $entryTimestamp = [DateTime]::ParseExact($matches[1], "dd-MM-yyyy HH:mm:ss:fff", $null)
                if ($entryTimestamp -gt $lastCheckTime) {
                    $newErrors += $entry
                }
            }
        }

        # Send alert if new errors exist
        if ($newErrors.Count -gt 0) {
            # Get the latest error timestamp to update our last check time
            $latestErrorTime = ($newErrors | ForEach-Object {
                if ($_ -match '^(\d{2}-\d{2}-\d{4} \d{2}:\d{2}:\d{2}:\d{3})') {
                    [DateTime]::ParseExact($matches[1], "dd-MM-yyyy HH:mm:ss:fff", $null)
                }
            } | Measure-Object -Maximum).Maximum

            # Compose and send email
            $emailSubject = $alertSubject -f $device
            $emailBody = $alertBodyTemplate -f $device, $lastCheckTime.ToString("yyyy-MM-dd HH:mm:ss"), ($newErrors -join "`n")
            Send-MailMessage -From $alertFrom -To $alertTo -Subject $emailSubject -Body $emailBody -SmtpServer $smtpServer

            # Update last check time to avoid repeat alerts
            $latestErrorTime.ToString("yyyy-MM-dd HH:mm:ss:fff") | Set-Content $lastCheckFile
            Write-Host "Alert sent! Found $($newErrors.Count) new errors on $device"
        } else {
            Write-Host "No new Invalid Data errors on $device since $lastCheckTime"
        }
    } catch {
        Write-Error "Failed to check $device : $_"
        # Optional: Send alert for unreachable device
        $errorSubject = "WARNING: Log check failed for $device"
        $errorBody = "Error accessing remote log file:`n`n$_"
        Send-MailMessage -From $alertFrom -To $alertTo -Subject $errorSubject -Body $errorBody -SmtpServer $smtpServer
    }
}

Optimizations for Hundreds of Devices #

Since you're managing hundreds of devices, here's how to scale this solution:

• Parallel Processing: Use PowerShell 7+'s ForEach-Object -Parallel to check multiple devices at once (avoids sequential bottlenecks):

  $remoteDevices | ForEach-Object -Parallel {
      $device = $_
      $targetLogPath = $using:remoteLogPath -f $device
      $lastCheckDir = $using:lastCheckDir
      # Repeat the single-device check logic here (use $using: prefix for external variables)
  } -ThrottleLimit 20 # Adjust based on your network capacity
  

• Efficient Log Reading: For large growing logs, instead of reading the entire file every time, track the byte position you last read from. This reduces I/O load:

  # Track read position instead of timestamp (add to per-device storage)
  $readPositionFile = Join-Path $lastCheckDir "$device`_position.txt"
  $lastReadPosition = if (Test-Path $readPositionFile) { [int64](Get-Content $readPositionFile) } else { 0 }
  
  $logFile = Get-Item $targetLogPath
  $newContent = Get-Content $logFile -Raw -ErrorAction Stop
  $contentBytes = [System.Text.Encoding]::UTF8.GetBytes($newContent)
  $newEntries = [System.Text.Encoding]::UTF8.GetString($contentBytes[$lastReadPosition..($contentBytes.Length-1)])
  
  # Process $newEntries for errors...
  
  # Update read position
  $logFile.Length | Set-Content $readPositionFile
  

• Centralized State Storage: Store last check timestamps/positions on a shared file server or SQLite database instead of local disk—this makes it easier to manage from multiple admin machines.

Adjusting to Frequent Checks #

When you shorten the check interval (e.g., to 5 minutes):

• Keep the timestamp/position tracking logic (it works regardless of interval)
• Avoid using Get-Content -Wait for remote files (it's unstable over SMB); stick to scheduled polling with Task Scheduler
• If you need near-real-time alerts, consider installing a lightweight monitoring agent on remote devices (but that's more complex than polling)

内容的提问来源于stack exchange，提问作者Nathan Tuck