PowerShell Invoke-WebRequest是否具备1000条URL定时巡检的可扩展性？

在装有PowerShell v5.1的单台Windows Server 2016虚拟机上，需要每5分钟从指定文本文件中检查1000个不同URL的可用性。最初计划用以下脚本实现：

$urlList = Get-Content -Path "c:\URLsList.txt"

foreach ($url in $urlList) {
    $result = Invoke-WebRequest $url
    $result.StatusCode
}

但因为URL数量多达1000条，不确定Invoke-WebRequest的扩展性，官方文档里没找到相关最佳实践或限制。了解到PowerShell Jobs，但不确定是否适合每5分钟一次的巡检需求。

1. 并行处理提升效率 #

单线程循环处理1000个URL会极慢，每个请求都要等待响应。PowerShell Jobs可实现并行，但更高效的是用RunspacePool（资源占用远低于Jobs），也可以用轻量的Invoke-RestMethod结合并行逻辑。

用RunspacePool实现高效并行 #

这是处理大量URL的最优方案，复用运行空间减少资源开销：

$urlList = Get-Content -Path "c:\URLsList.txt"
$maxThreads = 50 # 可根据服务器性能调整，建议20-100区间
$runspacePool = [RunspaceFactory]::CreateRunspacePool(1, $maxThreads)
$runspacePool.Open()
$jobs = @()

foreach ($url in $urlList) {
    $job = [PowerShell]::Create().AddScript({
        param($url)
        try {
            $response = Invoke-WebRequest -Uri $url -TimeoutSec 10 -ErrorAction Stop
            [PSCustomObject]@{
                URL = $url
                StatusCode = $response.StatusCode
                Status = "Success"
                Timestamp = Get-Date
            }
        }
        catch {
            [PSCustomObject]@{
                URL = $url
                StatusCode = $_.Exception.Response.StatusCode.value__
                Status = "Failed"
                ErrorMessage = $_.Exception.Message
                Timestamp = Get-Date
            }
        }
    }).AddArgument($url)
    $job.RunspacePool = $runspacePool
    $jobs += @{Job = $job; Handle = $job.BeginInvoke()}
}

# 收集执行结果
$results = @()
foreach ($job in $jobs) {
    $results += $job.Job.EndInvoke($job.Handle)
    $job.Job.Dispose()
}

$runspacePool.Close()
$runspacePool.Dispose()

# 输出或持久化结果
$results | Export-Csv -Path "c:\URLCheckResults.csv" -Append -NoTypeInformation

用PowerShell Jobs实现并行（适合简单场景） #

如果不想用RunspacePool，Jobs是更易理解的并行方案，但要注意控制并发数避免资源过载：

$urlList = Get-Content -Path "c:\URLsList.txt"
$batchSize = 50 # 每批启动50个任务，避免一次性创建1000个Jobs
$results = @()

for ($i=0; $i -lt $urlList.Count; $i += $batchSize) {
    $batchUrls = $urlList[$i..($i+$batchSize-1)]
    $jobs = foreach ($url in $batchUrls) {
        Start-Job -ScriptBlock {
            param($url)
            try {
                $response = Invoke-WebRequest -Uri $url -TimeoutSec 10 -ErrorAction Stop
                return [PSCustomObject]@{
                    URL = $url
                    StatusCode = $response.StatusCode
                    Status = "Success"
                    Timestamp = Get-Date
                }
            }
            catch {
                return [PSCustomObject]@{
                    URL = $url
                    StatusCode = if ($_.Exception.Response) { $_.Exception.Response.StatusCode.value__ } else { "No Response" }
                    Status = "Failed"
                    ErrorMessage = $_.Exception.Message
                    Timestamp = Get-Date
                }
            }
        } -ArgumentList $url
    }

    # 等待批处理完成并收集结果
    $results += $jobs | Wait-Job | Receive-Job
    $jobs | Remove-Job # 清理已完成的Jobs
}

# 保存结果
$results | Export-Csv -Path "c:\URLCheckResults.csv" -Append -NoTypeInformation

2. 替代Invoke-WebRequest：用轻量工具提升性能 #

Invoke-WebRequest功能全面但冗余，如果只需要检查可用性，推荐用更轻量的Invoke-RestMethod或.NET原生HttpClient：

用.NET HttpClient实现并行 #

HttpClient是性能最优的方案，适合大规模URL检测：

$urlList = Get-Content -Path "c:\URLsList.txt"
$maxThreads = 50
$httpClientHandler = New-Object System.Net.Http.HttpClientHandler
$httpClient = New-Object System.Net.Http.HttpClient($httpClientHandler)
$httpClient.Timeout = [System.TimeSpan]::FromSeconds(10)

$runspacePool = [RunspaceFactory]::CreateRunspacePool(1, $maxThreads)
$runspacePool.Open()
$jobs = @()

foreach ($url in $urlList) {
    $job = [PowerShell]::Create().AddScript({
        param($httpClient, $url)
        try {
            $response = $httpClient.GetAsync($url).Result
            [PSCustomObject]@{
                URL = $url
                StatusCode = [int]$response.StatusCode
                Status = if ($response.IsSuccessStatusCode) { "Success" } else { "Failed" }
                Timestamp = Get-Date
            }
        }
        catch {
            [PSCustomObject]@{
                URL = $url
                StatusCode = "Error"
                Status = "Failed"
                ErrorMessage = $_.Exception.Message
                Timestamp = Get-Date
            }
        }
    }).AddArgument($httpClient).AddArgument($url)
    $job.RunspacePool = $runspacePool
    $jobs += @{Job = $job; Handle = $job.BeginInvoke()}
}

$results = @()
foreach ($job in $jobs) {
    $results += $job.Job.EndInvoke($job.Handle)
    $job.Job.Dispose()
}

$runspacePool.Close()
$runspacePool.Dispose()
$httpClient.Dispose()
$httpClientHandler.Dispose()

$results | Export-Csv -Path "c:\URLCheckResults.csv" -Append -NoTypeInformation

3. 定时执行：用Windows任务计划程序 #

每5分钟执行一次巡检，最可靠的方式是用Windows任务计划程序：

• 创建新任务，触发器设置为每5分钟执行
• 操作选择“启动程序”，程序/脚本填powershell.exe，参数填-ExecutionPolicy Bypass -File "C:\Path\To\Your\CheckScript.ps1"
• 确保任务运行账户有访问URL和文件路径的权限

不推荐用PowerShell的Start-Sleep循环实现定时，一旦脚本中断就会停止巡检，任务计划程序的稳定性远高于脚本内循环。

内容的提问来源于stack exchange，提问作者edwio