集群Matlab API调用脚本异常自动重启监控脚本开发需求

Shell 监控重启脚本 #

这个脚本会持续检测指定screen中的Matlab进程，一旦发现进程终止，就清理旧screen并重启脚本。

#!/bin/bash

# 配置参数
MATLAB_SCRIPT="/path/to/your/matlab_script.m"  # 替换为你的Matlab脚本路径
SCREEN_NAME="matlab_api_runner"                # 自定义screen会话名称
MATLAB_CMD="matlab -nodisplay -r \"run('$MATLAB_SCRIPT'); exit;\""  # 集群上的Matlab启动命令，根据实际调整
LOG_FILE="/path/to/restart_log.txt"            # 重启日志记录路径

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
}

while true; do
    # 检查指定screen会话是否存在，且Matlab进程在运行
    SCREEN_EXISTS=$(screen -list | grep -q "$SCREEN_NAME" && echo 1 || echo 0)
    MATLAB_RUNNING=$(ps aux | grep "$MATLAB_SCRIPT" | grep -v grep | grep -v "$0" | wc -l)

    if [ $SCREEN_EXISTS -eq 0 ] || [ $MATLAB_RUNNING -eq 0 ]; then
        log "检测到Matlab脚本崩溃或screen会话不存在，开始重启..."
        
        # 终止残留的screen会话（如果存在）
        if [ $SCREEN_EXISTS -eq 1 ]; then
            screen -S "$SCREEN_NAME" -X quit
            log "已终止旧screen会话: $SCREEN_NAME"
            sleep 2  # 给系统时间清理进程
        fi

        # 创建新screen会话并启动Matlab脚本
        screen -dmS "$SCREEN_NAME" bash -c "$MATLAB_CMD"
        log "已创建新screen会话并重启Matlab脚本"
    fi

    sleep 60  # 每60秒检测一次，可根据需求调整
done

使用说明 #

1. 修改脚本中的配置参数为你的实际路径和命令
2. 给脚本添加执行权限：chmod +x monitor_matlab.sh
3. 后台运行脚本：nohup ./monitor_matlab.sh &

Python 监控重启脚本 #

如果更习惯用Python，以下是等价实现，支持进程检测、screen清理和重启：

import subprocess
import time
from datetime import datetime

# 配置参数
MATLAB_SCRIPT = "/path/to/your/matlab_script.m"
SCREEN_NAME = "matlab_api_runner"
MATLAB_CMD = f"matlab -nodisplay -r \"run('{MATLAB_SCRIPT}'); exit;\""
LOG_FILE = "/path/to/restart_log.txt"

def log(message):
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    log_line = f"[{timestamp}] {message}\n"
    with open(LOG_FILE, "a") as f:
        f.write(log_line)
    print(log_line.strip())

def check_screen_exists():
    try:
        result = subprocess.run(["screen", "-list"], capture_output=True, text=True)
        return SCREEN_NAME in result.stdout
    except Exception as e:
        log(f"检查screen会话出错: {str(e)}")
        return False

def check_matlab_running():
    try:
        result = subprocess.run(["ps", "aux"], capture_output=True, text=True)
        lines = result.stdout.splitlines()
        count = 0
        for line in lines:
            if MATLAB_SCRIPT in line and "grep" not in line and "monitor_matlab.py" not in line:
                count += 1
        return count > 0
    except Exception as e:
        log(f"检查Matlab进程出错: {str(e)}")
        return False

def kill_screen():
    try:
        subprocess.run(["screen", "-S", SCREEN_NAME, "-X", "quit"], capture_output=True, text=True)
        log(f"已终止旧screen会话: {SCREEN_NAME}")
        time.sleep(2)
    except Exception as e:
        log(f"终止screen会话出错: {str(e)}")

def start_new_screen():
    try:
        subprocess.run(["screen", "-dmS", SCREEN_NAME, "bash", "-c", MATLAB_CMD], capture_output=True, text=True)
        log("已创建新screen会话并重启Matlab脚本")
    except Exception as e:
        log(f"启动新screen会话出错: {str(e)}")

if __name__ == "__main__":
    log("Matlab监控脚本启动")
    while True:
        screen_exists = check_screen_exists()
        matlab_running = check_matlab_running()

        if not screen_exists or not matlab_running:
            log("检测到Matlab脚本崩溃或screen会话不存在，开始重启...")
            if screen_exists:
                kill_screen()
            start_new_screen()
        
        time.sleep(60)  # 每60秒检测一次，可按需调整

使用说明 #

1. 修改配置参数为你的实际路径和命令
2. 后台运行脚本：nohup python3 monitor_matlab.py &

额外建议 #

• 可调整检测间隔（sleep参数），比如缩短到30秒以更快响应崩溃
• 建议在Matlab脚本中加入API请求间隔控制（如每次调用后暂停几秒），从根源减少因频率限制导致的崩溃
• 定期查看日志文件，排查重启频率是否异常，判断是否需要优化API调用逻辑

内容的提问来源于stack exchange，提问作者Carlos Aguila