优化PowerShell日志导出脚本性能，解决内存占用过高问题

Great catch on the memory issue—storing every single log entry in an in-memory array is definitely the culprit here, especially if you're dealing with large volumes of logs. Let's walk through a couple of effective ways to fix this, plus some extra tweaks to make your script leaner and more reliable:

Approach 1: Batch-Write JSON Data (Avoid Full In-Memory Storage) #

Instead of collecting all logs in $allLogs first, we can write each page of logs to the file incrementally. The trick here is handling the JSON array syntax correctly (opening [, adding commas between batches, closing ] at the end).

Here's the modified script:

#Name: getLogs.ps1 
#Purpose: Script for exporting logs to a json file 
#Variables
$org = "<api url here>" 
$token = "<api token here>" 
$filePath = "D:\Downloads\logs\test2.json" 

# Simplify date formatting (no need for substring hacks!)
$today = Get-Date -Format "yyyy-MM-dd"
$fromTime = "${today}T00:00:00Z" -replace ":", "%3A"
$toTime = "${today}T23:59:59Z" -replace ":", "%3A"

# Initial API URI
$uri = "$org/api/v1/logs?until=$toTime&limit=20&sortOrder=DESCENDING&q=&since=$fromTime"

# Initialize the JSON file with the opening bracket
"[" | Out-File -FilePath $filePath -Encoding utf8
$firstBatch = $true

DO {
    # Use Invoke-RestMethod instead of Invoke-WebRequest for direct JSON parsing
    $response = Invoke-RestMethod -Headers @{"Authorization" = "SSWS $token"} -Method Get -Uri $uri
    $jsonBatch = $response | ConvertTo-Json -Depth 100

    # Handle commas between batches
    if (-not $firstBatch) {
        "," | Out-File -FilePath $filePath -Encoding utf8 -Append
    }
    # Write the batch without the outer array brackets (since we already added the main [])
    $jsonBatch.Substring(1, $jsonBatch.Length - 2) | Out-File -FilePath $filePath -Encoding utf8 -Append

    # Update pagination URI
    $linkHeader = $response.Headers.Link
    if ($linkHeader -and $linkHeader.EndsWith('rel="next"')) {
        $uri = $linkHeader.Split("<")[1].Split(">")[0]
        $firstBatch = $false
        Start-Sleep -m 1001 # Avoid rate limits
    } else {
        break
    }
} while ($true)

# Close the JSON array
"]" | Out-File -FilePath $filePath -Encoding utf8 -Append

Key Improvements Here: #

• No in-memory array: We write each batch to disk immediately, so memory usage stays low regardless of how many logs you pull.
• Simplified date formatting: Replaced messy substring calls with cleaner string concatenation and replacement.
• Invoke-RestMethod: More efficient than Invoke-WebRequest because it directly returns parsed JSON objects, skipping the ConvertFrom-Json step.
• Proper JSON syntax: We manually handle the outer array brackets and commas between batches to ensure the final file is valid JSON.

Approach 2: Use .NET StreamWriter for Even Lower Memory Overhead #

For even better memory efficiency (especially with extremely large datasets), use System.IO.StreamWriter to write directly to the file stream without converting entire batches to strings first.

#Name: getLogs.ps1 
#Purpose: Script for exporting logs to a json file 
#Variables
$org = "<api url here>" 
$token = "<api token here>" 
$filePath = "D:\Downloads\logs\test2.json" 

$today = Get-Date -Format "yyyy-MM-dd"
$fromTime = "${today}T00:00:00Z" -replace ":", "%3A"
$toTime = "${today}T23:59:59Z" -replace ":", "%3A"

$uri = "$org/api/v1/logs?until=$toTime&limit=20&sortOrder=DESCENDING&q=&since=$fromTime"

# Create a StreamWriter (auto-flushes to disk, minimal memory footprint)
$writer = [System.IO.StreamWriter]::new($filePath, $false, [System.Text.Encoding]::UTF8)
$writer.Write("[")
$firstEntry = $true

try {
    DO {
        $response = Invoke-RestMethod -Headers @{"Authorization" = "SSWS $token"} -Method Get -Uri $uri
        
        foreach ($log in $response) {
            if (-not $firstEntry) {
                $writer.Write(",")
            }
            # Convert individual log entry to JSON and write directly
            $jsonEntry = $log | ConvertTo-Json -Depth 100
            $writer.Write($jsonEntry)
            $firstEntry = $false
        }

        $linkHeader = $response.Headers.Link
        if ($linkHeader -and $linkHeader.EndsWith('rel="next"')) {
            $uri = $linkHeader.Split("<")[1].Split(">")[0]
            Start-Sleep -m 1001
        } else {
            break
        }
    } while ($true)
} finally {
    # Always close the writer to avoid file corruption
    $writer.Write("]")
    $writer.Close()
    $writer.Dispose()
}

Why This Is Better: #

• Per-entry writing: We convert and write each log entry one at a time, so memory usage stays constant (only holding one log entry at a time, plus minimal stream overhead).
• Guaranteed cleanup: The try/finally block ensures the stream is closed properly even if the script fails mid-run, preventing corrupted JSON files.

Extra Optimization Tips #

• Increase limit parameter: If the API allows, bump the limit from 20 to a higher value (like 100 or 500) to reduce the number of API calls. Just make sure you still stay within rate limits.
• Add error handling: Wrap API calls in try/catch blocks to handle timeouts or failed requests gracefully.
• Check rate limit headers: Instead of fixed Start-Sleep, read the X-Rate-Limit-Remaining and X-Rate-Limit-Reset headers to adjust your delay dynamically.

内容的提问来源于stack exchange，提问作者user1416876