Azure Functions中Polly重试日志间歇性未在App Insights显示的问题

我们在Windows Azure Functions中实现了如下Polly重试策略：

public async Task<T> ExecuteTaskWithRetryAsync<T>(int maxRetryAttempts, int delayBetweenFirstRetryInSeconds, string invocationIdentifier, Func<Task<T>> function)
{
    logger.LogInformation($"Executing {invocationIdentifier} with retries.");
    var retryPolicy = Policy
        .Handle<HttpRequestException>(ex => ex.StatusCode != HttpStatusCode.BadRequest && ex.StatusCode != HttpStatusCode.Unauthorized)
        .WaitAndRetryAsync(
        maxRetryAttempts,
        retryAttempt => TimeSpan.FromSeconds(delayBetweenFirstRetryInSeconds * Math.Pow(2, retryAttempt - 1)),
        (exception, timeSpan, retryCount, context) =>
        {
            logger.LogWarning("Execution of {identifier} failed. Waiting {sec} seconds before retry. Attempt  number {retryCount}. Exception - {exceptionMessage}", invocationIdentifier, timeSpan.Seconds, retryCount, exception.Message);
        });
    return await retryPolicy.ExecuteAsync<T>(async () =>
    {
        T result = await function();
        logger.LogInformation($"Successfully executed {invocationIdentifier}");
        return result;
    });
}

其调用方式如下：

var response = await retryService.ExecuteTaskWithRetryAsync(retryOptions.MaxRetryAttempts, retryOptions.DelayBetweenRetriesInSeconds, nameof(GetFromMetaRPAsync), async () =>
{
    return await httpClient.GetAsync(endpoint).ConfigureAwait(false);
});
if (!response.IsSuccessStatusCode)
{
    throw new HttpRequestException($"Failed to retrieve any results. Response with status code {response?.StatusCode} - {response?.ReasonPhrase}.");
}

遇到的问题 #

• 在App Insights中仅能看到初始日志：

  "Executing {invocationIdentifier} with retries"

• 随后会抛出状态码为504（GatewayTimeout）的HttpRequestException，但在服务器出现BadGateway、GatewayTimeout等问题时，无法看到重试日志：

  "Waiting {sec} seconds before retry."

• 仅偶尔能看到如下日志行：

  "Waiting {sec} seconds before retry... Exception - There was error sending the request"

• 未捕获到重试日志的案例中，调用响应约20秒后才返回

疑问与验证需求 #

• 是否存在重试实际执行但未被App Insights捕获的可能？
• 即使依赖项未被记录，重试日志是否仍应被捕获？
• 如何验证重试是否实际发生？
• 已在本地运行代码并完成单元测试，两种场景下均能看到重试执行，云端行为是否可能存在差异？

内容的提问来源于stack exchange，提问作者Kruti Joshi