CentOS VPS中Shell脚本因内存超限终止后未写入日志的问题

在CentOS的VPS上运行作为安全WebSocket（WSS）服务器的JAR包，当内存占用超过850MB（VPS总内存1GB）时，系统会终止WSS服务。但当前的wss_service.sh脚本在系统主动终止进程后，不会写入*** JAR Crashed ***等后续日志内容；而通过Ctrl+C或htop手动终止进程时，日志则能正常生成。

现有脚本wss_service.sh #

#!/bin/bash
check=true
while [ "$check" = true ]
do
    startS=$(date +%s)
    dates=$(date '+%d-%m-%Y_%H-%M-%S')
    name="WSS_ServerJar_"
    new="$name$dates"
    files="./Log/$new.log"
    touch $files
    echo $files >> $files 2>&1
    echo "*** WSS Server started ***" >> $files 2>&1
    java -jar WSS_Server.jar >> $files 2>&1
    sleep 5
    echo "*** JAR Crashed ***" >> $files 2>&1
    endS=$(date +%s)
    diff=$(($endS-$startS))
    echo "diff=$diff; startS=$startS; endS=$endS;" >> $files 2>&1
    pass=120
    if (( pass > diff ));
    then
        check=false
        echo "Failed in short time: $diff" >> $files
    fi

done

系统终止进程后的日志内容 #

./Log/WSS_ServerJar_28-07-2023_01-35-13.log
*** WSS Server started ***
2023-07-28 01:35:14.874 [main] INFO  Application - Autoreload is disabled because the development mode is off.
Inside Application
2023-07-28 01:35:15.982 [main] INFO  Application - Application started in 1.162 seconds.
2023-07-28 01:35:16.655 [main] WARN  i.netty.util.internal.MacAddressUtil - Failed to find a usable hardware address from the network interfaces; using random bytes: a5:f2:64:ba:ab:26:51:3f
2023-07-28 01:35:16.823 [DefaultDispatcher-worker-2] INFO  Application - Responding at https://0.0.0.0:1234
2023-07-28 03:46:32.491 [KtorShutdownHook] TRACE io.ktor.server.websocket.WebSockets - Shutdown WebSockets due to application stop
2023-07-28 03:46:32.494 [main] TRACE io.ktor.server.websocket.WebSockets - Shutdown WebSockets due to application stop

手动终止进程后的日志内容 #

./Log/WSS_ServerJar_28-07-2023_13-48-13.log
*** WSS Server started ***
2023-07-28 13:48:14.628 [main] INFO  Application - Autoreload is disabled because the development mode is off.
Inside Application
2023-07-28 13:48:15.515 [main] INFO  Application - Application started in 0.971 seconds.
2023-07-28 13:48:16.101 [main] WARN  i.netty.util.internal.MacAddressUtil - Failed to find a usable hardware address from the network interfaces; using random bytes: c8:9d:92:76:47:73:29:09
2023-07-28 13:48:16.291 [DefaultDispatcher-worker-2] INFO  Application - Responding at https://0.0.0.0:1234
2023-07-28 13:48:27.859 [KtorShutdownHook] TRACE io.ktor.server.websocket.WebSockets - Shutdown WebSockets due to application stop
2023-07-28 13:48:27.860 [main] TRACE io.ktor.server.websocket.WebSockets - Shutdown WebSockets due to application stop
*** JAR Crashed ***
diff=21; startS=1690532293; endS=1690532314;
Failed in short time: 21

提问：为什么系统主动终止进程时，*** JAR Crashed ***等后续内容没有写入日志？

核心原因 #

系统因内存不足终止进程时，触发的是OOM Killer（内存不足终止器），它会直接向目标进程发送SIGKILL强制终止信号：

1. SIGKILL无法被进程捕获、忽略或处理，bash脚本中java -jar命令后的代码，只有在进程正常退出或收到可捕获信号（如Ctrl+C发送的SIGINT）时才会执行；
2. 内存极度紧张时，bash进程自身可能也没有足够内存执行后续的echo、date等命令，导致日志写入失败；
3. 极端情况下，OOM Killer可能连带终止bash脚本进程，直接中断后续逻辑。

解决方案 #

1. 捕获可处理的终止信号 #

给脚本添加信号捕获逻辑，确保收到SIGTERM、SIGINT等可处理信号时能正常写入日志，在脚本开头添加：

# 定义信号处理函数
handle_shutdown() {
    echo "*** JAR Terminated by Signal ***" >> "$files" 2>&1
    endS=$(date +%s)
    diff=$(($endS-$startS))
    echo "diff=$diff; startS=$startS; endS=$endS;" >> "$files" 2>&1
    exit 1
}

# 捕获SIGINT和SIGTERM信号
trap handle_shutdown SIGINT SIGTERM

2. 改用系统日志服务记录事件 #

将日志写入操作交给系统日志服务（如syslog），减少bash脚本自身内存占用，即使脚本内存不足也能大概率记录事件：

# 替换原有日志写入方式
echo "*** WSS Server started ***" | logger -t WSS_Server
java -jar WSS_Server.jar 2>&1 | logger -t WSS_Server
echo "*** JAR Crashed ***" | logger -t WSS_Server

通过journalctl -t WSS_Server即可查看所有日志内容。

3. 监控内存并主动重启 #

在脚本中加入内存监控，当占用接近阈值时主动重启JAR包，避免被OOM Killer强制终止：

while [ "$check" = true ]
do
    startS=$(date +%s)
    dates=$(date '+%d-%m-%Y_%H-%M-%S')
    name="WSS_ServerJar_"
    new="$name$dates"
    files="./Log/$new.log"
    touch $files
    echo $files >> $files 2>&1
    echo "*** WSS Server started ***" >> $files 2>&1
    
    # 后台启动JAR并记录PID
    java -jar WSS_Server.jar >> $files 2>&1 &
    JAR_PID=$!
    
    # 每分钟检查一次内存
    while kill -0 $JAR_PID 2>/dev/null; do
        MEM_USAGE=$(free | awk '/Mem/{printf("%.0f", $3/$2*100)}')
        if [ $MEM_USAGE -ge 80 ]; then
            echo "*** Memory usage reached $MEM_USAGE%, restarting JAR ***" >> $files 2>&1
            kill $JAR_PID
            break
        fi
        sleep 60
    done
    
    sleep 5
    echo "*** JAR Crashed or Restarted ***" >> $files 2>&1
    endS=$(date +%s)
    diff=$(($endS-$startS))
    echo "diff=$diff; startS=$startS; endS=$endS;" >> $files 2>&1
    pass=120
    if (( pass > diff ));
    then
        check=false
        echo "Failed in short time: $diff" >> $files
    fi

done

4. 限制JVM内存占用 #

给Java进程设置最大堆内存，避免占用过多内存触发OOM Killer：

java -Xmx768m -jar WSS_Server.jar >> $files 2>&1

-Xmx768m表示将JVM最大堆内存限制为768MB，预留足够内存给系统和其他进程。

内容的提问来源于stack exchange，提问作者Cool_Coder