能否为不同用途配置同一logstash.conf？附Spring Boot场景示例

Absolutely! You can absolutely use a single logstash.conf to handle distinct log types like regular application logs and error stack traces—especially with your Spring Boot + logstash-logback-encoder setup. Here's a practical, maintainable approach tailored to your stack:

Step 1: Tag Logs at the Source (Logback) #

First, we'll add explicit metadata to your logs directly in Logback so Logstash can easily distinguish between log types. The logstash-logback-encoder lets you inject custom fields via the customFields parameter:

<!-- logback-spring.xml -->
<!-- Application Log Appender: Tags regular app logs -->
<appender name="LOGSTASH_APP" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
  <destination>localhost:5044</destination> <!-- Match Logstash's input port -->
  <encoder class="net.logstash.logback.encoder.LogstashEncoder">
    <customFields>{"log_type":"application"}</customFields>
  </encoder>
</appender>

<!-- Error Stack Appender: Tags error/stack trace logs -->
<appender name="LOGSTASH_ERROR" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
  <destination>localhost:5044</destination>
  <encoder class="net.logstash.logback.encoder.LogstashEncoder">
    <customFields>{"log_type":"error_stack"}</customFields>
  </encoder>
</appender>

<!-- Route logs to the right appenders -->
<root level="INFO">
  <appender-ref ref="LOGSTASH_APP"/> <!-- Send all INFO+ logs to app appender -->
</root>

<!-- Send ERROR-level logs (including stacks) to the error appender -->
<logger name="com.yourcompany" level="ERROR" additivity="false">
  <appender-ref ref="LOGSTASH_ERROR"/>
</logger>

Note: The additivity="false" flag prevents error logs from being duplicated across both appenders.

Step 2: Conditional Processing in Logstash Config #

Now, use the log_type field to build conditional logic in your single logstash.conf to handle each log type differently:

# logstash.conf
input {
  tcp {
    port => 5044
    codec => json_lines # Matches the JSON output from logstash-logback-encoder
  }
}

filter {
  # Handle regular application logs
  if [log_type] == "application" {
    mutate {
      remove_field => ["@version", "host"] # Clean up unused fields
      add_field => ["processing_context", "application_log"]
    }
    # Optional: Extract MDC fields (e.g., request IDs from Spring)
    if [mdc][requestId] {
      mutate {
        rename => ["[mdc][requestId]", "request_id"] # Flatten MDC field for easier querying
      }
    }
  }

  # Handle error stack traces
  else if [log_type] == "error_stack" {
    mutate {
      add_field => ["processing_context", "error_stack"]
    }
    # Parse exception details from stack traces (logstash-logback-encoder auto-populates [stack_trace])
    if [stack_trace] {
      grok {
        match => {"stack_trace" => "^%{DATA:exception_class}: %{GREEDYDATA:exception_message}"}
        tag_on_failure => [] # Don't tag failures for non-standard stack formats
      }
    }
  }
}

output {
  # Route to different Elasticsearch indexes (or files, Kafka, etc.)
  if [log_type] == "application" {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "app-logs-%{+YYYY.MM.dd}"
    }
  } else if [log_type] == "error_stack" {
    elasticsearch {
      hosts => ["localhost:9200"]
      index => "error-logs-%{+YYYY.MM.dd}"
    }
    # Optional: Backup error stacks to a file
    file {
      path => "/var/log/logstash/error-stacks-%{+YYYY.MM.dd}.log"
      codec => json_lines
    }
  }
}

Alternative: No Logback Appender Split #

If you don't want to split appenders in Logback, you can still use Logstash conditionals based on log level or content:

• Use if [level] == "ERROR" to target error logs
• Use if "Exception" in [message] to detect stack traces (though less reliable than explicit tagging)

But tagging at the source (Logback) is cleaner and more maintainable, especially as your log volume grows.

内容的提问来源于stack exchange，提问作者Alessio Frabotta