Bash脚本处理Minecraft日志过慢？是特性还是代码可优化？

近期需要处理Minecraft服务器日志，流程为解压多个gzip压缩日志、合并后提取有效信息。最初编写的Bash脚本执行耗时60秒，改用Python实现仅需9秒。随后对Bash脚本逐步优化：移除嵌套循环后耗时降至23秒，删除冗余变量后耗时进一步降到5秒。这引发疑问：到底是Bash本身性能差，还是初始写法存在问题？

初始Bash脚本（耗时60秒） #

dir="/home/minecraft/spigot-1.20.2/logs"

logs=$(ls $dir)

start=$SECONDS
sum=""

for log in $logs
do
    s=$SECONDS
    log_date=$(echo $log | cut -b 1-10)
    unzipped_log="$(zcat -f $dir/$log | sed -e "s/\[Async Chat Thread - #[0-9]\+\/INFO]: //" -e "s/^/$log_date /g")"$'\n'
    while read -r line
    do
        echo "$line"
    done <<< "$unzipped_log"
    echo The file $log $(du -h $dir/$log) processed in $(($SECONDS - $s))
done

s=$SECONDS
last="$(cat $dir/latest.log | sed -e "s/\[Async Chat Thread - #[0-9]\+\/INFO]: //")"$'\n'
date_today=$(date +%Y-%m-%d)
while read -r line
do
    echo "$date_today $line"
done <<< "$last"
echo The file latest.log processed in $(($SECONDS - $s))

echo Script worked in $(($SECONDS-$start)) seconds

执行命令：source sum_logs.sh | tac > all_logs.txt

Python实现脚本（耗时9秒） #

import os, gzip, time, re
from datetime import datetime

dir = "/home/minecraft/spigot-1.20.2/logs"

start = time.time()
time.sleep(1)

logs = os.listdir(dir)

for log in logs:
    log_path = os.path.join(dir, log)
    log_date = log[:10]
    try:
        with gzip.open(log_path, 'rt') as f:
            for line in f.readlines():
                line = re.sub(r'\[Async Chat Thread - #\d+\/INFO]: ', '', line)
                print(log_date, line.replace('\n', ''))
    except:
        with open(log_path, 'r') as f:
            for line in f.readlines():
                line = re.sub(r'\[Async Chat Thread - #\d+\/INFO]: ', '', line)
                print(log_date, line.replace('\n', ''))

elapsed = round(time.time() - start)

print(f'Script worked in {elapsed} seconds')

第一次优化后的Bash脚本（耗时23秒） #

dir="/home/minecraft/spigot-1.20.2/logs"

start=$SECONDS
sum=""

for f in $dir/*
do
    s=$SECONDS
    log=$(echo $f | cut -b 36-)
    log_date=$(echo $log | cut -b 1-10)
    unzipped_log="$(zcat -f $dir/$log | sed -e "s/\[Async Chat Thread - #[0-9]\+\/INFO]: //" -e "s/^/$log_date /g")"$'\n'
    echo "$unzipped_log"
    echo The file $log $(du -h $dir/$log) processed in $(($SECONDS - $s))
done

s=$SECONDS
date_today=$(date +%Y-%m-%d)
last="$(cat $dir/latest.log | sed -e "s/\[Async Chat Thread - #[0-9]\+\/INFO]: //" -e "s/^/$date_today /g")"$'\n'
echo $last
echo The file latest.log processed in $(($SECONDS - $s))

echo Script worked in $(($SECONDS-$start)) seconds

最终优化后的Bash脚本（耗时5秒） #

dir="/home/minecraft/spigot-1.20.2/logs"

start=$SECONDS
sum=""

for f in $dir/*
do
    s=$SECONDS
    log=$(echo $f | cut -b 36-)
    log_date=$(echo $log | cut -b 1-10)
    zcat -f $dir/$log | sed -e "s/\[Async Chat Thread - #[0-9]\+\/INFO]: //" -e "s/^/$log_date /g"
    echo The file $log $(du -h $dir/$log) processed in $(($SECONDS - $s))
done

s=$SECONDS
date_today=$(date +%Y-%m-%d)
cat $dir/latest.log | sed -e "s/\[Async Chat Thread - #[0-9]\+\/INFO]: //" -e "s/^/$date_today /g"
echo The file latest.log processed in $(($SECONDS - $s))

echo Script worked in $(($SECONDS-$start)) seconds

核心结论：初始写法低效是主因，Bash并非天生性能差 #

• 初始脚本的致命问题：
  1. 错误遍历文件：用logs=$(ls $dir)获取文件列表，不仅在文件名含空格时会出错，还会额外创建子进程，增加开销。
  2. 全量加载+逐行echo：把整个日志内容读到unzipped_log变量中，再通过while read循环逐行echo，每一行都会触发一个新的echo子进程，这是最耗时的操作。
  3. 冗余嵌套循环：明明可以直接输出zcat | sed的结果，却多套了一层循环，完全没必要。
• 优化后的性能提升逻辑：
  1. 移除嵌套循环后，直接输出处理结果，避免了大量子进程创建，耗时从60秒降到23秒。
  2. 最终优化去掉了将日志内容存入变量的步骤，让zcat和sed通过管道直接流式输出，不需要加载整个文件到内存，耗时进一步降至5秒——甚至比Python版本更快，因为zcat和sed都是C语言编写的高效工具，流式处理效率远超Python逐行读取。
• Bash的性能特性：
  Bash作为脚本语言，启动子进程的开销确实比Python大，但只要合理利用Unix管道和内置命令，避免不必要的子进程和内存加载，性能完全可以和Python媲美甚至超越。

内容的提问来源于stack exchange，提问作者upconett