容器化ASP.NET服务通过Grafana Loki收集日志失败求助

本地IDE调试时，ASP.NET服务通过Serilog可正常将日志发送至Loki，但容器化部署后无法发送。尝试配置Promtail采集日志，仅能获取容器内/var/lib/log/dpkg.log路径下的系统日志，且容器内未生成预期的日志文件夹。

相关配置 #

Docker-compose文件 #

version: "3.3"
networks:
  loki:
services:
  myservice.api:
    image: myservice.api
    build:
      context: ../
      dockerfile: ./Source/Services/myservice/myservice.api/Dockerfile
    container_name: myservice
    ports:
      - "${MYSERVICE_API_PORT}"
    environment:
      - ${MYSERVICE_API_URLS}
    networks:
      - loki
  loki:
      container_name: loki
      image: grafana/loki:2.4.0
      volumes:
        - ./Monitoring/config/loki:/etc/loki
      ports:
        - "3100:3100"
      restart: unless-stopped
      command: -config.file=/etc/loki/local-config.yaml
      networks:
        - loki

  grafana:
      container_name: grafana
      image: grafana/grafana:latest
      user: "1000"
      volumes:
        - ./Monitoring/config/grafana:/etc/lib/grafana
      ports:
        - "3000:3000"
      environment:
        - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
        - GF_AUTH_ANONYMOUS_ENABLED=true
        - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      entrypoint:
        - sh
        - -euc
        - |
          mkdir -p /etc/grafana/provisioning/datasources
          cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
          apiVersion: 1
          datasources:
          - name: Loki
            type: loki
            access: proxy
            orgId: 1
            url: http://loki:3100
            basicAuth: false
            isDefault: true
            version: 1
            editable: true
            apiVersion: 1
          EOF
          /run.sh
      restart: unless-stopped
      networks:
        - loki

Loki配置 #

auth_enabled: false

server:
    http_listen_address: 0.0.0.0
    http_listen_port: 3100
    grpc_listen_port: 9096

common: 
    path_prefix: /tmp/loki
    storage: 
        filesystem:
            chunks_directory: /tmp/loki/chunks
            rules_directory: /tmp/loki/rules
    replication_factor: 1
    ring: 
        instance_addr: 127.0.0.1
        kvstore:
            store: inmemory

schema_config:
    configs:
        - from: 2023-08-22
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
          index: 
            prefix: index_
            period: 24h

ruler: 
    alertmanager_url: http://localhost:9093

应用代码片段 #

Setup.cs

public class Setup
{
    /// <summary>
    /// Configure logger
    /// </summary>
    /// <returns>Configured logger</returns>
    public static LoggerConfiguration GetLoggerConfiguration()
    {
        return new LoggerConfiguration()
            .WriteTo.Console(LogEventLevel.Debug)
            .WriteTo.File(
                new JsonFormatter(),
                "../logs/log-.json",
                LogEventLevel.Fatal,
                rollingInterval: RollingInterval.Day)
            .WriteTo.GrafanaLoki(
                "http://host.docker.internal:3100",
                new List<LokiLabel>()
                {
                    new()
                    {
                        Key = "Service",
                        Value = System.IO.Path.GetFileName(Assembly.GetExecutingAssembly().Location).Replace(".dll", "")
                    }
                });
    }
}

Program.cs

public class Program
{
    public static void Main(string[] args)
    {   
        Log.Logger = Setups.GetLoggerConfiguration().CreateLogger();
        CreateHostBuilder(args).Build().Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args)
    {
        return Host.CreateDefaultBuilder(args)
                .ConfigureLogging((context, logging) =>
                {
                    logging.SetMinimumLevel(LogLevel.Critical);
                    logging.ClearProviders();
                })
                .UseSerilog()
                .ConfigureAppConfiguration((hostingContext, builder) =>
                {
                    var hostingEnvironment = hostingContext.HostingEnvironment;
                    builder
                        .AddJsonFile("appsettings.json", true, true)
                        .AddJsonFile($"appsettings.{hostingEnvironment.EnvironmentName}.json", true)
                        .AddJsonFile("appsettings.local.json", true)
                        .AddEnvironmentVariables();
                        
                    if (args == null)
                    {
                        return;
                    }

                    builder.AddCommandLine(args);
                })
                .ConfigureWebHostDefaults(webBuilder => { webBuilder.UseStartup<Startup>(); }
                );
    }
}

解决方案 #

1. 修正Serilog连接Loki的地址 #

容器在自定义Docker网络中，需通过服务名loki而非host.docker.internal访问Loki（后者用于容器访问宿主机）。修改Setup.cs中的Loki地址：

.WriteTo.GrafanaLoki(
    "http://loki:3100", // 替换原http://host.docker.internal:3100
    new List<LokiLabel>()
    {
        new()
        {
            Key = "Service",
            Value = System.IO.Path.GetFileName(Assembly.GetExecutingAssembly().Location).Replace(".dll", "")
        }
    });

2. 修复容器内日志文件生成问题 #

• 本地相对路径../logs/log-.json在容器内无效，需改用绝对路径，同时调整日志级别（原Fatal级别过低，难以触发）：

  .WriteTo.File(
      new JsonFormatter(),
      "/app/logs/log-.json", // 容器内绝对路径
      LogEventLevel.Information, // 调整为Information级别
      rollingInterval: RollingInterval.Day)
  

• 在ASP.NET服务的Dockerfile中添加创建日志目录的命令，确保路径存在：

  RUN mkdir -p /app/logs
  

3. 调整Promtail配置采集应用日志 #

当前Promtail仅采集系统日志，需新增配置采集ASP.NET容器日志，同时修正Loki地址：

server:
  http_listen_port: 9080
  grpc_listen_port: 0
  
clients:
  - url: http://loki:3100/loki/api/v1/push # 替换docker.host.internal为loki

positions:
  filename: /tmp/positions.yaml
  
target_config:
  sync_period: 10s
  
scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log
  # 新增ASP.NET服务日志采集
  - job_name: aspnet-service
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 10s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/myservice' # 匹配你的容器名
        action: keep
      - source_labels: ['__meta_docker_container_log_path']
        target_label: '__path__'
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'

在docker-compose.yml中添加Promtail服务，挂载必要的资源：

promtail:
  image: grafana/promtail:2.4.0
  volumes:
    - ./Monitoring/config/promtail:/etc/promtail
    - /var/run/docker.sock:/var/run/docker.sock
    - /var/lib/docker/containers:/var/lib/docker/containers:ro
  command: -config.file=/etc/promtail/config.yaml
  networks:
    - loki

4. 验证容器网络连通性 #

进入ASP.NET容器，测试与Loki的连通性：

docker exec -it myservice curl http://loki:3100/ready

返回ready则网络正常；若不通，检查所有服务是否加入loki网络。

内容的提问来源于stack exchange，提问作者GasTorrent