Azure Function因NextLink分页卡住问题求助

问题描述 #

我创建了一个每日运行的Azure Timer Trigger Function，用于获取所有AAD组成员。测试运行时，日志显示NextLink分页迭代几次后无报错卡住，已将函数超时时间调整至10分钟仍未解决。本地运行该脚本一切正常，预期需迭代127次、获取12757条数据，但Azure监控日志中无法找到卡住的会话。

原代码如下：

using namespace System.Net

# Input bindings are passed in via param block.
param($Timer)

"START"

$fileName = "Azure - ADGroupMembers.csv"

Write-Output "Connect to AAD"
Connect-AzAccount -identity

try {
    #region auth

   Write-Output "Get items from MS Graph"
   $token = (Get-AzAccessToken -ResourceUrl "https://graph.microsoft.com/").Token
   $authHeader = @{Authorization = "Bearer $token"}
    
    # #endregion
    
    # #region main proces

    $allPages = @()
    $items = (Invoke-RestMethod -Method 'Get' -Uri 'https://graph.microsoft.com/v1.0/groups?$top=999&$expand=members' -Headers $authHeader  -ContentType 'Application/Json') 
    $allPages += $items.value

    $index = 0
    if ($items.'@odata.nextLink') {
            do {
                  $index++
                  "Index counter: $index"

                  $token = (Get-AzAccessToken -ResourceUrl "https://graph.microsoft.com/").Token
                  $authHeader = @{Authorization = "Bearer $token"}

                  $items = (Invoke-RestMethod -Method 'Get' -Uri $items.'@odata.nextLink' -Headers $authHeader  -ContentType 'Application/Json')
                  $allPages += $items.value
             } until (
                 !$items.'@odata.nextLink'
             )
    }

    $items = $allPages
    "Count items: " + $items.Count

    $filePath = "D:\home\data\$($fileName)"

    Write-Host "Convert to csv to path '$($filePath)'" -ForegroundColor green
    $items | Export-Csv -NoTypeInformation -Path $filePath
    
    
    #endregion
}
catch {
     write-output $_.Exception.Message
}

"FINISH"

可能原因 #

• 请求无超时限制：Invoke-RestMethod默认没有超时配置，当Graph API响应缓慢时，请求会无限挂起直到函数超时。
• Graph API节流未处理：大规模分页请求触发Graph API的429节流机制，函数未做重试处理，导致请求卡住。
• 内存溢出：$allPages数组持续累加数据，Azure函数内存配额不足时，进程会无响应而非抛出明确错误。
• Token异常未捕获：虽然每次循环重新获取token，但函数环境中可能存在token缓存或获取失败，导致后续请求无效却未报错。

解决方案（修改后的代码） #

针对上述问题，优化后的代码加入超时控制、节流重试、内存优化和详细日志：

using namespace System.Net

# Input bindings are passed in via param block.
param($Timer)

Write-Output "START: $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')"

$fileName = "Azure - ADGroupMembers.csv"
$tempFilePath = "D:\home\data\Temp_$fileName"
$finalFilePath = "D:\home\data\$fileName"

Write-Output "Connect to AAD via managed identity"
Connect-AzAccount -Identity

try {
    Write-Output "Initialize Graph API auth header"
    $token = (Get-AzAccessToken -ResourceUrl "https://graph.microsoft.com/").Token
    $authHeader = @{Authorization = "Bearer $token"}

    $pageIndex = 0
    $totalItems = 0
    $firstPage = $true

    # 初始请求
    $response = Invoke-RestMethod -Method 'Get' -Uri 'https://graph.microsoft.com/v1.0/groups?$top=999&$expand=members' -Headers $authHeader -ContentType 'Application/Json' -TimeoutSec 300
    $totalItems += $response.value.Count
    Write-Output "Page $pageIndex : $($response.value.Count) items, total: $totalItems"

    # 写入临时文件（避免内存溢出）
    if ($firstPage) {
        $response.value | Export-Csv -NoTypeInformation -Path $tempFilePath
        $firstPage = $false
    } else {
        $response.value | Export-Csv -NoTypeInformation -Path $tempFilePath -Append
    }

    while ($response.'@odata.nextLink') {
        $pageIndex++
        Write-Output "Processing page $pageIndex, NextLink: $($response.'@odata.nextLink')"

        # 重新获取token避免过期
        $token = (Get-AzAccessToken -ResourceUrl "https://graph.microsoft.com/").Token
        $authHeader = @{Authorization = "Bearer $token"}

        try {
            $response = Invoke-RestMethod -Method 'Get' -Uri $response.'@odata.nextLink' -Headers $authHeader -ContentType 'Application/Json' -TimeoutSec 300 -ErrorAction Stop
            $totalItems += $response.value.Count
            Write-Output "Page $pageIndex : $($response.value.Count) items, total: $totalItems"

            # 追加到临时文件
            $response.value | Export-Csv -NoTypeInformation -Path $tempFilePath -Append
        } catch {
            if ($_.Exception.Response.StatusCode -eq 429) {
                # 处理Graph API节流，按提示重试
                $retryAfter = [int]$_.Exception.Response.Headers['Retry-After']
                Write-Output "Throttled by Graph API, retrying after $retryAfter seconds..."
                Start-Sleep -Seconds $retryAfter
                $pageIndex-- # 重试当前页，不递增索引
            } else {
                throw $_ # 其他错误抛出到外层catch
            }
        }
    }

    Write-Output "Total items retrieved: $totalItems"
    Write-Output "Moving temp file to final path: $finalFilePath"
    Move-Item -Path $tempFilePath -Destination $finalFilePath -Force

} catch {
    # 捕获完整异常信息便于排查
    Write-Output "ERROR: $($_.Exception.ToString())"
    # 清理临时文件
    if (Test-Path $tempFilePath) {
        Remove-Item $tempFilePath -Force
    }
}

Write-Output "FINISH: $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')"

关键修改点 #

• 给Invoke-RestMethod添加-TimeoutSec参数，避免请求无限挂起
• 处理Graph API 429节流状态码，自动按Retry-After提示重试
• 采用临时文件追加写入，避免一次性加载所有数据导致内存溢出
• 增加时间戳、分页详情、错误堆栈等日志，便于定位问题
• 捕获完整异常信息，而非仅错误消息

验证步骤 #

1. 部署修改后的函数，将超时时间设置为15分钟以上（根据数据量调整）
2. 查看函数日志，确认每页都能正常处理直至完成所有分页
3. 检查输出文件的记录数量是否符合预期
4. 若仍有问题，查看函数的内存使用指标，确认是否需要调高内存配额

内容的提问来源于stack exchange，提问作者KEM