如何动态集成Azure Log Analytics告警与对应Application Insights追踪

问题背景 #

我们同时使用Log Analytics Workspace和Application Insights监控应用程序，已在Log Analytics中创建带自定义负载的告警规则（如检测失败请求）。当告警触发时，希望动态获取并关联Application Insights中的对应追踪数据，附加到告警通知以辅助分析。

已完成工作 #

• 创建Log Analytics告警规则：
  • 定义KQL查询以捕获特定日志
  • 设置告警逻辑并配置动作组
• 考虑集成方案：
  • 使用Azure Logic Apps自动化关联流程
  • 探索Azure Functions以编程方式查询和处理数据

核心需求 #

我们需要详细的解决方案或示例，展示如何：

• 解析告警负载以提取相关详情
• 根据告警详情查询Application Insights获取关联追踪数据
• 发送关联数据（如通过邮件、Teams/Slack或作为DevOps工作项）

现有示例代码 #

Log Analytics告警查询 #

AppRequests
| where success == false
| summarize count() by bin(timestamp, 5m)

Azure Function示例代码（C#） #

using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Extensions.Logging;
using Microsoft.Rest.Azure.Authentication;
using Microsoft.Azure.OperationalInsights;
using Microsoft.Azure.ApplicationInsights.Query;

public static class AlertFunction
{
    [FunctionName("LogAnalyticsAlert")]
    public static async Task<HttpResponseMessage> Run(
        [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequestMessage req,
        ILogger log)
    {
        var alertPayload = await req.Content.ReadAsAsync<AlertPayload>();
        var logAnalyticsClient = new OperationalInsightsDataClient(new TokenCredentials("your-token"));
        var appInsightsClient = new ApplicationInsightsDataClient(new TokenCredentials("your-token"));

        var logsQuery = $"traces | where timestamp between (datetime({alertPayload.StartTime}) .. datetime({alertPayload.EndTime}))";
        var appInsightsResults = await appInsightsClient.Query.ExecuteAsync("your-app-insights-id", logsQuery);

        // Process and correlate results

        return req.CreateResponse(HttpStatusCode.OK, appInsightsResults);
    }
}

完整解决方案实现 #

1. 解析告警负载提取关键信息 #

首先定义AlertPayload类匹配Log Analytics告警的JSON结构，核心字段覆盖时间范围、触发日志的原始数据：

public class AlertPayload
{
    public string SchemaId { get; set; }
    public Data Data { get; set; }
}

public class Data
{
    public string SearchQuery { get; set; }
    public string StartTime { get; set; }
    public string EndTime { get; set; }
    public Table[] Tables { get; set; }
}

public class Table
{
    public string Name { get; set; }
    public Column[] Columns { get; set; }
    public string[][] Rows { get; set; }
}

public class Column
{
    public string Name { get; set; }
    public string Type { get; set; }
}

解析注意事项：

• 从Data.Tables中提取告警触发的原始日志字段（比如失败请求的operation_Id）
• 将StartTime和EndTime转换为UTC时间，确保与Application Insights的时间戳对齐

2. 查询Application Insights关联追踪数据 #

基于提取的失败请求operation_Id，编写精准KQL查询获取对应追踪日志、异常信息：

// 从告警负载中提取失败请求的operation_Id集合
var failedOperations = new List<string>();
foreach (var row in alertPayload.Data.Tables[0].Rows)
{
    // 索引需匹配你的KQL查询结果列顺序，示例假设第3列是operation_Id
    var opId = row[2];
    failedOperations.Add(opId);
}

// 构建关联查询，聚合多类型追踪数据
var aiQuery = $@"
union traces, exceptions, requests
| where operation_Id in ({string.Join(",", failedOperations.Select(id => $"'{id}'"))})
| where timestamp between (datetime({alertPayload.Data.StartTime}) .. datetime({alertPayload.Data.EndTime}))
| project timestamp, operation_Id, message, type, severityLevel
| order by timestamp desc
";

// 执行查询（建议用托管标识认证，替代硬编码token）
var aiClient = new ApplicationInsightsDataClient(new TokenCredentials(await GetManagedIdentityToken()));
var aiResults = await aiClient.Query.ExecuteAsync("your-app-insights-resource-id", aiQuery);

// 格式化查询结果为可读文本
var formattedResults = string.Join("\n\n", aiResults.Tables[0].Rows.Select(row => 
    $"时间: {row[0]}\n操作ID: {row[1]}\n内容: {row[2]}\n类型: {row[3]}"
));

3. 发送包含关联数据的告警通知 #

以Teams消息为例，通过Webhook发送格式化后的告警内容：

// 构建Teams消息卡片
var teamsPayload = new
{
    @type = "MessageCard",
    themeColor = "FF0000",
    title = "应用失败请求告警",
    text = $"检测到{failedOperations.Count}个失败请求，关联追踪数据如下:\n\n{formattedResults}",
    sections = new[]
    {
        new {
            facts = new[]
            {
                new { name = "告警时间范围", value = $"{alertPayload.Data.StartTime} 至 {alertPayload.Data.EndTime}" }
            }
        }
    }
};

// 发送请求到Teams Webhook
using var httpClient = new HttpClient();
await httpClient.PostAsync(
    "your-teams-webhook-url",
    new StringContent(JsonConvert.SerializeObject(teamsPayload), Encoding.UTF8, "application/json")
);

最佳实践 #

• 使用托管标识认证：为Azure Function分配Log Analytics和Application Insights的读取权限，通过Azure AD托管标识获取访问令牌，避免硬编码凭证
• 优化KQL查询性能：缩小时间范围至告警触发窗口，用operation_Id精准过滤，避免全量扫描
• 复用客户端实例：在Function中缓存OperationalInsightsDataClient和ApplicationInsightsDataClient，减少初始化开销
• 添加错误处理：捕获查询失败、通知发送失败等异常，记录日志并设置重试机制
• 监控集成流程：在Azure Function中添加自定义日志，通过Application Insights监控函数执行状态

内容的提问来源于stack exchange，提问作者Vowneee