Serilog无法向ElasticSearch写入日志的问题排查求助

我当前使用Docker Compose、ASP.NET Core 6、Elastic Stack技术栈，Elastic Stack已正常运行，Serilog能在控制台输出日志，但无法将日志写入ElasticSearch。

使用的NuGet包 #

• Serilog.AspNetCore
• Serilog.Enrichers.Environment
• Serilog.Exceptions
• Serilog.Settings.Configuration
• Serilog.Sinks.Debug
• Serilog.Sinks.Elasticsearch

Docker Compose配置 #

version: '3.4'

networks:
    mediatrexample:
        driver: bridge
    elk_stack:
        driver: bridge

services:
    pgsql_image:
        image: postgres:latest
        container_name: postgersql_db
        restart: always
        environment:
            POSTGRES_USER: "postgres"
            POSTGRES_PASSWORD: "123456"
            POSTGRES_DB: "ecommerce"
        ports:
            - 5432:5432
        networks:
            - mediatrexample

    redis_image:
        image: 'redis:latest'
        container_name: redis_db
        restart: always
        environment:
            - REDIS_PASSWORD=RRnFPZ93tjBHB9W62p
            - REDIS_PORT=6379
        ports:
            - '6379:6379'
        command: [ "redis-server" ]
        networks:
            - mediatrexample

    elastic_image:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.16.1
        container_name: elk_elastic
        environment:
            - node.name=elastic01
            - xpack.monitoring.enabled=true
            - xpack.watcher.enabled=false
            - xpack.security.enabled=true
            - xpack.security.audit.enabled=true
            - discovery.type=single-node
            - ELASTIC_PASSWORD=DkIedPPSCb
            - ES_JAVA_OPTS=-Xms512m -Xmx512m
        ulimits:
            memlock:
                soft: -1
                hard: -1
        networks:
            - elk_stack
        ports:
            - 9200:9200
            - 9300:9300

    kibana_image:
        image: docker.elastic.co/kibana/kibana:7.16.1
        container_name: elk_kibana
        environment:
            - ELASTICSEARCH_URL=http://elastic_image:9200
            - ELASTICSEARCH_HOSTS=http://elastic_image:9200
            - ELASTICSEARCH_USERNAME=elastic
            - ELASTICSEARCH_PASSWORD=DkIedPPSCb
        networks:
            - elk_stack
        depends_on:
            - elastic_image
        links:
            - elastic_image
        ports:
            - 5601:5601

    mediatrexample.api:
        image: ${DOCKER_REGISTRY-}mediatrexampleapi
        container_name: mediatr_example
        depends_on:
            - pgsql_image
            - redis_image
            - elastic_image
            - kibana_image
        build:
            context: .
            dockerfile: MediatrExample.API/Dockerfile
        networks:
            - mediatrexample
        ports:
            - 5001:80 
        links:
            - pgsql_image
            - redis_image
            - elastic_image
        environment:
            - CONNECTIONSTRINGS__NpgSQLConnection=User ID=postgres;Password=123456;Host=pgsql_image;Port=5432;Database=ecommerce
            - Redis__Host=redis_image
            - Redis__Port=6379
            - Redis__Password=RRnFPZ93tjBHB9W62p
            #- Serilog__WriteTo__1__Args__nodeUris=http://elastic:DkIedPPSCb@elastic_image:9200

日志配置代码（MyLogger类） #

public static class MyLogger
{
    public static void ConfigureSeriLog(IConfiguration configuration)
    {
        string elasticUser = configuration["SeriLogConfig:ElasticUser"];
        string elasticPassword = configuration["SeriLogConfig:ElasticPassword"];

        Log.Logger = new LoggerConfiguration()
            .MinimumLevel.Debug()
            .Enrich.FromLogContext()
            .Enrich.WithMachineName()
            .WriteTo.Debug()
            .WriteTo.Console()
            .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(configuration["SeriLogConfig:ElasticUri"]))
            {
                AutoRegisterTemplate = true,
                AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
                IndexFormat = $"{configuration["SeriLogConfig:ProjectName"]}-{configuration["SeriLogConfig:Environment"]}-logs-{0:yyyy.MM.dd}",
                ModifyConnectionSettings = s => s.BasicAuthentication(elasticUser, elasticPassword),
                FailureCallback = e => { Console.WriteLine("Hasan - Unable to submit event -- " + e.MessageTemplate); e.Properties.Values.ToList().ForEach(x => Console.Write(x)); },
                EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
                               EmitEventFailureHandling.WriteToFailureSink |
                               EmitEventFailureHandling.RaiseCallback |
                               EmitEventFailureHandling.ThrowException,
                FailureSink = new FileSink("./failures.txt", new JsonFormatter(), null)
            })
            .Enrich.WithProperty("Environment", configuration["SeriLogConfig:Environment"])
            .ReadFrom.Configuration(configuration, "SerilogLogLevels")
            .CreateLogger();
    }
}

Program.cs代码 #

var builder = WebApplication.CreateBuilder(args);
var Configuration = builder.Configuration;
// Add services to the container.

MyLogger.ConfigureSeriLog(Configuration);
builder.Host.UseSerilog();

builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
    .AddJwtBearer(JwtBearerDefaults.AuthenticationScheme, opt =>
    {
        opt.TokenValidationParameters = new TokenValidationParameters()
        {
            IssuerSigningKey = new SymmetricSecurityKey(Encoding.UTF8.GetBytes(Configuration["JWTTokenOptions:SecretKey"])),
            ValidateLifetime = true,
            ClockSkew = TimeSpan.FromMinutes(1), //Skew default 5 minute
            ValidateAudience = false,
            ValidateIssuer = false,
            ValidateIssuerSigningKey = true
        };
    });

builder.Services.AddControllers();

var multiplexer = RedisUtility.ConnectRedis(Configuration);
builder.Services.AddSingleton<IConnectionMultiplexer>(multiplexer);

builder.Services.AddMediatR(Assembly.GetAssembly(typeof(GetAllUserRequest)));
builder.Services.TryAddSingleton<IHttpContextAccessor, HttpContextAccessor>();
builder.Services.AddMyServiceLifeCycles(Configuration);
builder.Services.AddValidatorsFromAssembly(Assembly.GetAssembly(typeof(GetAllUserRequestValidator)));
builder.Services.ConfigureMyOptionModels(Configuration);

builder.Services.AddEntityFrameworkNpgsql().AddDbContext<AppDbContext>(opt =>
{
    opt.UseNpgsql(Configuration.GetConnectionString("NpgSQLConnection"), npsqlOpt =>
    {
        npsqlOpt.CommandTimeout(150);
        npsqlOpt.MigrationsAssembly("MediatrExample.Data");
    });
});
// Above Code for "timestamp with time zone' literal cannot be generated for Local DateTime: a UTC DateTime is required" error solution
AppContext.SetSwitch("Npgsql.EnableLegacyTimestampBehavior", true);

builder.Services.AddCors(options => { options.AddPolicy("AllowOrigin", policy => policy.AllowAnyOrigin()); });

var app = builder.Build();

app.UseMyCustomMiddleware();

app.UseCors(builder => builder.AllowAnyOrigin().AllowAnyMethod().AllowAnyHeader());

app.UseHttpsRedirection();

app.UseRouting();

app.UseAuthentication();
app.UseAuthorization();

app.MapControllers();

app.MapGet("/", () =>
{
    using (var scope = app.Services.CreateAsyncScope())
    {
        var service = scope.ServiceProvider;
        var logger = service.GetRequiredService<ILogger<Program>>();
        logger.LogInformation("Example Log Date: {0}", DateTime.Now);
    }
    return "Server is Running...";
});

#region Auto Run Pending Migration
using (var scope = app.Services.CreateAsyncScope())
{
    var services = scope.ServiceProvider;
    var dbContext = services.GetRequiredService<AppDbContext>();
    if (dbContext.Database.GetPendingMigrations().Any())
    {
        dbContext.Database.Migrate();
    }
}
#endregion

app.Run();

appsettings.json配置 #

{
  "ConnectionStrings": {
    "NpgSQLConnection": "User ID=postgres;Password=123456;Host=pgsql_image;Port=5432;Database=ecommerce;Integrated Security=true;Pooling=true;"
  },
  "Environment": {
    "EnvironmentName": "dev"
  },
  "Redis": {
    "Host": "0.0.0.0",
    "Port": "6380",
    "Password": "123456"
  },
  "JWTTokenOptions": {
    "SecretKey": "v5PSdoOHjeB34wUkmMwr4x4Z1HWkudFoyVZTx7sFvm",
    "TokenExpireTimeMinute": 120
  },
  "Logging": {
    "LogLevel": {
      "Default": "Debug",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "Serilog": {
    "Using": [ "Serilog.Sinks.Console", "Serilog.Sinks.Elasticsearch", "Serilog.Enrichers.Environment" ],
    "MinimumLevel": {
      "Default": "Debug",
      "Override": {
        "Microsoft": "Information",
        "System": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "Console"
      },
      {
        "Name": "Elasticsearch",
        "Args": {
          "nodeUris": "http://localhost:9200",
          "indexFormat": "mediatr-example-logs-{0:yyyy.MM.dd}",
          "autoRegisterTemplate": true,
          "autoRegisterTemplateVersion": "ESv7",
          "DetectElasticsearchVersion": true,
          "OverwriteTemplate": true,
          "emitEventFailure": "WriteToSelfLog",
          "registerTemplateFailure": "IndexAnyway",
          "numberOfShards": 20,
          "numberOfReplicas": 10,
          "connectionGlobalHeaders": "Authorization=Basic ZWxhc3RpYzpEa0llZFBQU0Ni"
        }
      }
    ],
    "Enrich": [ "FromLogContext", "WithMachineName" ]
  },
  "SerilogLogLevels": {
    "MinimumLevel": {
      "Default": "Debug",
      "Override": {
        "Microsoft": "Information",
        "System": "Warning"
      }
    }
  },
  "SeriLogConfig": {
    "ProjectName": "mediatr-example",
    "ElasticUri": "http://localhost:9200",
    "Environment": "development",
    "ElasticUser": "elastic",
    "ElasticPassword": "DkIedPPSCb"
  },
  "AllowedHosts": "*"
}

已尝试的配置方式 #

• 在MyLogger.ConfigureSeriLog方法中硬编码配置Serilog
• 在appsettings.json中通过配置文件定义Serilog

核心排查与修复方向 #

1. Docker跨网络访问问题
   ASP.NET Core容器属于mediatrexample网络，ElasticSearch容器属于elk_stack网络，跨网络默认无法互通。需将api容器加入elk_stack网络：

   # 在mediatrexample.api的networks节点添加
   networks:
       - mediatrexample
       - elk_stack
   

   同时，容器内访问ElasticSearch不能用localhost，需改用服务名elastic_image，将appsettings.json中SeriLogConfig:ElasticUri和Serilog:WriteTo:1:Args:nodeUris都修改为http://elastic_image:9200。

2. 配置冲突与优先级问题
   同时使用代码硬编码和配置文件两种方式初始化Serilog，容易导致配置覆盖。建议统一方式：

   • 用代码配置：移除配置文件中Serilog节点，确保代码中所有ElasticSearch配置正确指向容器服务名。
   • 用配置文件：删除MyLogger.ConfigureSeriLog()调用，替换为：

     builder.Host.UseSerilog((context, config) => config.ReadFrom.Configuration(context.Configuration));
     

3. 索引格式与ElasticSearch参数问题

   • 代码中IndexFormat的时间占位符错误，应改为{Timestamp:yyyy.MM.dd}，否则会导致索引名称无效：

     IndexFormat = $"{configuration["SeriLogConfig:ProjectName"]}-{configuration["SeriLogConfig:Environment"]}-logs-{Timestamp:yyyy.MM.dd}"
     

   • 单节点ElasticSearch设置20分片、10副本不合理，会导致索引创建失败，将appsettings.json中numberOfShards改为1，numberOfReplicas改为0。

4. 查看失败日志定位具体原因
   代码中配置了FailureCallback和FailureSink，可进入api容器查看./failures.txt文件，或查看容器控制台输出的错误信息，直接定位网络不通、认证失败、索引创建失败等具体问题。

内容的提问来源于stack exchange，提问作者filiphasan