如何在命令行消费带人类可读时间戳的Kafka消息？

Got it, let's walk through the practical ways to get human-readable timestamps when using the kafka-console-consumer tool. The built-in print.timestamp=true spits out epoch time by default, but we can fix that with a couple of straightforward approaches depending on your Kafka version:

Method 1: Use Built-in Formatter (Kafka 2.1.0+) #

Starting from Kafka 2.1.0, the DefaultMessageFormatter supports a print.timestamp.format property that lets you directly output timestamps in a human-friendly format. This is the cleanest approach since it doesn't rely on external tools.

Here's a complete command example:

kafka-console-consumer.sh --bootstrap-server your-kafka-broker:9092 \
  --topic your-target-topic \
  --from-beginning \
  --formatter org.apache.kafka.tools.DefaultMessageFormatter \
  --property print.timestamp=true \
  --property print.key=true \
  --property print.value=true \
  --property timestamp.type=CreateTime \
  --property print.timestamp.format="yyyy-MM-dd HH:mm:ss.SSS"

Key Details: #

• timestamp.type: Pick between CreateTime (the timestamp when the message was produced) or LogAppendTime (when Kafka wrote the message to the broker).
• print.timestamp.format: Uses Java's SimpleDateFormat syntax, so you can tweak it to your needs (e.g., yyyy-MM-dd for just the date, or HH:mm:ss for time only).
• Include print.key=true/print.value=true only if you need to see the message key and value alongside the timestamp.

Method 2: Pipe to Shell Tools (Older Kafka Versions) #

If you're stuck on a Kafka version before 2.1.0 (where print.timestamp.format doesn't exist), you can pipe the epoch timestamp output to awk and date to convert it on the fly.

For Linux (GNU Date): #

kafka-console-consumer.sh --bootstrap-server your-kafka-broker:9092 \
  --topic your-target-topic \
  --from-beginning \
  --formatter org.apache.kafka.tools.DefaultMessageFormatter \
  --property print.timestamp=true \
  --property print.key=true \
  --property print.value=true | awk -F'\t' '{ 
    cmd="date -d @"$1" +\"%Y-%m-%d %H:%M:%S.%3N\""; 
    cmd | getline human_ts; 
    close(cmd); 
    print human_ts "\t" $2 "\t" $3 
  }'

For macOS (BSD Date): #

The date syntax is slightly different here:

kafka-console-consumer.sh --bootstrap-server your-kafka-broker:9092 \
  --topic your-target-topic \
  --from-beginning \
  --formatter org.apache.kafka.tools.DefaultMessageFormatter \
  --property print.timestamp=true \
  --property print.key=true \
  --property print.value=true | awk -F'\t' '{ 
    cmd="date -r "$1" +\"%Y-%m-%d %H:%M:%S.%3N\""; 
    cmd | getline human_ts; 
    close(cmd); 
    print human_ts "\t" $2 "\t" $3 
  }'

Notes for This Method: #

• The default output delimiter from DefaultMessageFormatter is a tab (\t), so we use -F'\t' in awk to split the fields. If you've changed the delimiter with property key.separator, adjust this accordingly.
• %3N adds millisecond precision (matching the epoch timestamp's precision from Kafka). If you don't need milliseconds, you can remove that part.

Quick Pre-Checks: #

• Verify your Kafka version with kafka-console-consumer.sh --version to pick the right method.
• If some messages show null for the timestamp, it means those messages were produced without a timestamp (common in very old Kafka setups).

内容的提问来源于stack exchange，提问作者petertc