如何使用awk命令过滤保留最新7天内的多段日志条目

需要过滤日志文件，移除比最新条目早7天的内容。例如最新日志日期为2024-02-13时，2024-02-05及更早的日志需全部删除。

示例日志文件 #

* Server Name: myserver
* Date and Time: 2024-02-05 23:00:01
* Total Number of Child (PID): 4
* PID: 117703, Worker Threads: 46
* PID: 117704, Worker Threads: 30
* PID: 117705, Worker Threads: 30
* PID: 117809, Worker Threads: 30

* Server Name: myserver
* Date and Time: 2024-02-05 23:30:01
* Total Number of Child (PID): 4
* PID: 117703, Worker Threads: 46
* PID: 117704, Worker Threads: 30
* PID: 117705, Worker Threads: 30
* PID: 117809, Worker Threads: 30

* Server Name: myserver
* Date and Time: 2024-02-06 00:00:01
* Total Number of Child (PID): 4
* PID: 117703, Worker Threads: 46
* PID: 117704, Worker Threads: 30
* PID: 117705, Worker Threads: 30
* PID: 117809, Worker Threads: 30

.
.
.

* Server Name: myserver
* Date and Time: 2024-02-13 23:00:01
* Total Number of Child (PID): 4
* PID: 117703, Worker Threads: 46
* PID: 117704, Worker Threads: 30
* PID: 117705, Worker Threads: 30
* PID: 117809, Worker Threads: 30

* Server Name: myserver
* Date and Time: 2024-02-13 23:30:01
* Total Number of Child (PID): 4
* PID: 117703, Worker Threads: 46
* PID: 117704, Worker Threads: 30
* PID: 117705, Worker Threads: 30
* PID: 117809, Worker Threads: 30

期望输出 #

* Server Name: myserver
* Date and Time: 2024-02-06 00:00:01
* Total Number of Child (PID): 4
* PID: 117703, Worker Threads: 46
* PID: 117704, Worker Threads: 30
* PID: 117705, Worker Threads: 30
* PID: 117809, Worker Threads: 30

.
.
.

* Server Name: myserver
* Date and Time: 2024-02-13 23:00:01
* Total Number of Child (PID): 4
* PID: 117703, Worker Threads: 46
* PID: 117704, Worker Threads: 30
* PID: 117705, Worker Threads: 30
* PID: 117809, Worker Threads: 30

* Server Name: myserver
* Date and Time: 2024-02-13 23:30:01
* Total Number of Child (PID): 4
* PID: 117703, Worker Threads: 46
* PID: 117704, Worker Threads: 30
* PID: 117705, Worker Threads: 30
* PID: 117809, Worker Threads: 30

原脚本问题 #

我编写了以下bash脚本，但运行后直接清空了日志文件，未达到预期效果：

#!/bin/bash

LOG_FILE="logs/worker_count.log"
TMP_FILE="$LOG_FILE.tmp"

# Extract the newest and oldest dates from the log file
newest_date=$(grep -oP 'Date and Time: \K[^\n]+' "$LOG_FILE" | tail -n 1)
oldest_date=$(grep -oP 'Date and Time: \K[^\n]+' "$LOG_FILE" | head -n 1)

# Check if either date is empty
if [ -z "$newest_date" ] || [ -z "$oldest_date" ]; then
    echo "Error: Unable to extract dates from the log file."
    exit 1
fi

# Convert dates to timestamps for comparison
newest_timestamp=$(date -d "$newest_date" +"%s")
oldest_timestamp=$(date -d "$oldest_date" +"%s")

# Calculate the difference in seconds
time_difference=$((newest_timestamp - oldest_timestamp))

# If the difference is greater than or equal to 7 days (604800 seconds)
if [ "$time_difference" -ge 604800 ]; then
        # Calculate the cutoff date based on the newest date minus 7 days
        cutoff_date=$(date -d "@$((newest_timestamp - 604800))" +"%Y-%m-%d %T")

        # Extract entries within the specified date range and remove old entries
        awk -v cutoff="$cutoff_date" '/^(\* Server Name:|\* Date and Time:)/ {
        server_name = $NF
        getline datetime
        if (datetime >= cutoff) {
          print "* Server Name: " server_name
          print datetime
          for (i = 1; i <= 5; i++) {
            getline line
            print line
          }
        }
      }
    ' "$LOG_FILE" > "$TMP_FILE"

        # Replace the original log file with the filtered entries
        #mv "$TMP_FILE" "$LOG_FILE"
else
        echo "No need to remove old entries. Time difference is less than 7 days."
fi

原脚本核心问题 #

1. getline使用逻辑错误：正则同时匹配Server Name和Date and Time行，导致重复读取行，破坏日志条目结构，最终跳过有效内容或读取空行。
2. 日期比较逻辑失效：datetime变量读取的是带* Date and Time:前缀的整行，拿这个字符串和纯日期格式的cutoff_date比较，结果永远为假，导致临时文件无内容输出。
3. 硬编码条目行数：用固定循环读取5行后续内容，若日志条目格式变化（行数增减），会导致内容错位。

修正后的脚本 #

#!/bin/bash

LOG_FILE="logs/worker_count.log"
TMP_FILE="$LOG_FILE.tmp"

# 提取最新日志日期并计算7天前的截止时间戳
newest_date=$(grep -oP 'Date and Time: \K[^\n]+' "$LOG_FILE" | tail -n 1)
if [ -z "$newest_date" ]; then
    echo "Error: Unable to extract newest date from log file."
    exit 1
fi

newest_ts=$(date -d "$newest_date" +"%s")
cutoff_ts=$((newest_ts - 7*24*3600))

# 使用awk处理日志，按条目分割并过滤
awk -v cutoff="$cutoff_ts" '
    # 以空行作为日志条目的分隔符
    BEGIN { RS = ""; ORS = "\n\n" }
    {
        # 从当前条目中提取日期时间字符串
        if (match($0, /Date and Time: ([0-9-]+ [0-9:]+)/, m)) {
            # 将日期转换为时间戳
            cmd = "date -d \"" m[1] "\" +%s"
            cmd | getline ts
            close(cmd)
            # 保留时间戳大于等于截止值的条目
            if (ts >= cutoff) {
                print $0
            }
        }
    }
' "$LOG_FILE" > "$TMP_FILE"

# 检查临时文件是否为空，避免覆盖原日志
if [ -s "$TMP_FILE" ]; then
    mv "$TMP_FILE" "$LOG_FILE"
    echo "Old log entries removed successfully."
else
    echo "No valid entries after filtering, keeping original log."
    rm "$TMP_FILE"
fi

脚本说明 #

• 按条目分割处理：用RS = ""将空行作为日志条目分隔符，每个条目作为整体处理，避免逐行读取的混乱。
• 可靠的日期比较：将提取的日期字符串转换为时间戳，用数值比较替代字符串比较，结果更准确。
• 动态适配条目格式：无需硬编码条目行数，直接输出符合条件的完整条目，适配格式变化。
• 安全覆盖机制：检查临时文件非空后再替换原日志，避免误清空文件。

内容的提问来源于stack exchange，提问作者Megan