如何用Bash按cost字段数值对Hadoop日志进行排序

需要从日志中提取cost:xxxms格式里的数值部分，并根据该数值对整条日志行进行排序。

日志示例 #

/var/log/hadoop/hdfs/hadoop-hdfs-datanode-datanode01.star.com.log.7:2025-04-24 11:56:57,334 WARN  datanode.DataNode (BlockReceiver.java:receivePacket(701)) - Slow BlockReceiver write data to disk cost:377ms (threshold=300ms), volume=/data/sdg/hadoop/hdfs/data
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-datanode01.star.com.log.7:2025-04-24 11:56:57,334 WARN  datanode.DataNode (BlockReceiver.java:receivePacket(701)) - Slow BlockReceiver write data to disk cost:507ms (threshold=300ms), volume=/data/sdg/hadoop/hdfs/data
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-datanode01.star.com.log.7:2025-04-24 11:56:57,334 WARN  datanode.DataNode (BlockReceiver.java:receivePacket(701)) - Slow BlockReceiver write data to disk cost:337ms (threshold=300ms), volume=/data/sdg/hadoop/hdfs/data
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-datanode01.star.com.log.7:2025-04-24 11:56:57,334 WARN  datanode.DataNode (BlockReceiver.java:receivePacket(701)) - Slow BlockReceiver write data to disk cost:407ms (threshold=300ms), volume=/data/sdg/hadoop/hdfs/data

期望输出 #

/var/log/hadoop/hdfs/hadoop-hdfs-datanode-datanode01.star.com.log.7:2025-04-24 11:56:57,334 WARN  datanode.DataNode (BlockReceiver.java:receivePacket(701)) - Slow BlockReceiver write data to disk cost:337ms (threshold=300ms), volume=/data/sdg/hadoop/hdfs/data
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-datanode01.star.com.log.7:2025-04-24 11:56:57,334 WARN  datanode.DataNode (BlockReceiver.java:receivePacket(701)) - Slow BlockReceiver write data to disk cost:377ms (threshold=300ms), volume=/data/sdg/hadoop/hdfs/data
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-datanode01.star.com.log.7:2025-04-24 11:56:57,334 WARN  datanode.DataNode (BlockReceiver.java:receivePacket(701)) - Slow BlockReceiver write data to disk cost:407ms (threshold=300ms), volume=/data/sdg/hadoop/hdfs/data
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-datanode01.star.com.log.7:2025-04-24 11:56:57,334 WARN  datanode.DataNode (BlockReceiver.java:receivePacket(701)) - Slow BlockReceiver write data to disk cost:507ms (threshold=300ms), volume=/data/sdg/hadoop/hdfs/data

尝试过的失败命令 #

grep 'cost:' log_file.txt | awk -F'cost:' '{print $0, $2}' | awk -F'ms' '{print $0, $1}' | sort -t' ' -k2,2nr
grep 'cost:' log_file.txt | awk -F'cost:' '{gsub("ms", "", $2); print $0, $2}' | sort -t' ' -k2,2nr
grep 'cost:' log_file.txt | awk -F'cost:' '{gsub("ms", "", $2); print $0, $2}' | sort -t' ' -k2,2nr | cut -d' ' -f1-

解决方案 #

使用以下命令可以实现需求：

grep 'cost:' log_file.txt | awk '{match($0, /cost:([0-9]+)ms/, arr); print arr[1], $0}' | sort -n | cut -d' ' -f2-

命令说明 #

1. grep 'cost:' log_file.txt：过滤出所有包含cost:的日志行；
2. awk '{match($0, /cost:([0-9]+)ms/, arr); print arr[1], $0}'：通过正则匹配提取cost:后的纯数值，将数值放在行首，后面拼接原日志行；
3. sort -n：按行首的数值从小到大排序（如需降序，添加-r参数，即sort -nr）；
4. cut -d' ' -f2-：去掉行首的排序数值，仅保留原日志行内容。

失败原因说明 #

之前的命令失效是因为原日志行包含大量空格，用空格作为分隔符指定排序列时，无法准确定位到提取的cost数值所在的位置，导致排序逻辑出错。

内容的提问来源于stack exchange，提问作者King David