如何用Logstash解析日期并将指定格式日志转为目标JSON结构

Hey there! Let's get your Logstash setup dialed in to parse your logs exactly how you want. Below is a full, tested configuration that will extract all the fields you need, handle timestamp conversion, and output the structure you specified.

Step 1: Full Working Configuration File #

input {
  file {
    path => "/path/to/your/target/logfile.log" # Replace with your actual log file path
    start_position => "beginning"
    sincedb_path => "/dev/null" # Use this to reprocess the entire file on each run (great for testing)
  }
}

filter {
  # 1. Parse raw log line into individual fields with Grok
  grok {
    match => { "message" => "%{DATE:timestamp_raw} %{TIME:time_raw} %{WORD:_method1} %{WORD:_method2} :: %{GREEDYDATA:line}" }
    remove_field => ["message", "timestamp_raw", "time_raw"] # Clean up temporary helper fields
  }

  # 2. Build your custom timestamp field and sync Logstash's @timestamp
  mutate {
    add_field => { "timestamp" => "%{timestamp_raw} %{time_raw}" }
  }

  date {
    match => [ "timestamp", "yyyy/MM/dd HH:mm:ss" ]
    target => "@timestamp"
    timezone => "UTC" # Adjust to your local timezone if needed (e.g., "Europe/Paris")
  }

  # 3. Set the required _type field
  mutate {
    add_field => { "_type" => "logs" }
  }
}

output {
  # For testing: Print parsed events to console in readable format
  stdout {
    codec => rubydebug
  }

  # Uncomment below if you want to send parsed logs to Elasticsearch
  # elasticsearch {
  #   hosts => ["localhost:9200"]
  #   index => "application-logs-%{+YYYY.MM.dd}"
  # }
}

Step 2: Key Details of Each Section #

• Input: The file input reads your log file directly. Update the path to point to your actual log location. The sincedb_path setting ensures Logstash reprocesses the entire file every time you run it, which is handy for testing changes.
• Grok Filter: This breaks down your log line into meaningful fields: it extracts the date/time parts, _method1, _method2, and the full log message into line. We then clean up the temporary helper fields we no longer need.
• Timestamp Handling: We first combine the date and time parts into your custom timestamp field. The date filter converts this value into Logstash's standard @timestamp (required for time-based filtering in tools like Kibana) while keeping your original timestamp field intact as requested.
• Type Field: The final mutate filter adds the _type field set to "logs" to match your desired output structure.

Step 3: Test the Configuration #

1. Save the configuration to a file (e.g., logstash-log-parser.conf).
2. Check for syntax errors first:

   bin/logstash -f logstash-log-parser.conf --config.test_and_exit
   

3. Run Logstash to process your logs:

   bin/logstash -f logstash-log-parser.conf
   

Expected Output Structure #

You'll get an event that matches your requested format:

{
  "_type": "logs",
  "_method1": "SYST",
  "_method2": "DEBUG",
  "line": "RefOPoolConnexionsSQL::getConnexionSQL() --> A016",
  "_source": {
    "path": "/path/to/your/target/logfile.log",
    "@timestamp": "2018-03-15T16:22:31.000Z", # Logstash's standard ISO 8601 timestamp
    "timestamp": "2018/03/15 16:22:31" # Your custom timestamp
  }
}

Note: Logstash's @timestamp uses ISO 8601 format (UTC by default) which is the industry standard for time-series data in tools like Elasticsearch. If you need it to match your custom timestamp format exactly, you can add an extra mutate filter to reformat it, but keeping the standard format is recommended for time-based operations.

内容的提问来源于stack exchange，提问作者issam zebdi