PBS任务遇孤立Docker容器提前终止，脚本执行不全求助

问题背景 #

有一个每5分钟运行一次的应用，采用Docker+PBS调度执行，核心为Python脚本。多数时候运行正常，但偶尔PBS清理**孤立容器（orphaned container）**时，会导致任务提前终止，底层Python脚本仅部分执行。

现有PBS执行脚本 #

#PBS -l select=1:ncpus=8:mem=40G,walltime=00:10:00
#PBS -N regmod_realtime_forecast
#PBS -o output_regmod_realtime_forecast.log
#PBS -e error_regmod_realtime_forecast.log
#PBS -q Hercules
#PBS -v CONTAINER_IMAGE=rm_test_202402
#PBS -p 100
#MANDATORY
source ~/.bashrc
####

cd /data/regional_model/src
python _realtime_forecast.py

validation_result=$(python ~/scripts/regmod_forecast_validation.py)

if [ "$validation_result" == "False" ]; then
        python _realtime_forecast.py
fi

异常与正常执行日志对比 #

异常执行日志（触发孤立容器清理时） #

**05/01/2024 12:30:26;0080;pbs_python;Hook;pbs_python;Cleaning up orphaned container: "373189.oracle"**
05/01/2024 12:30:27;0100;pbs_python;Hook;PBS_hpc_container;argv new: /opt/pbs/sbin/pbs_container 373189.oracle 0 /opt/pbs exec -w /home/seradmin 373189.oracle /bin/bash -c /var/spool/pbs/mom_priv/jobs/373189.oracle.SC
05/01/2024 12:30:27;0008;pbs_mom;Job;373189.oracle;Started, pid = 259270
05/01/2024 12:30:28;0008;pbs_mom;Job;373189.oracle;pid 260117 sid 260117 cmd 373189.orac attached as task 00000002
05/01/2024 12:30:28;0001;pbs_mom;Job;373189.oracle;tm_attach: sid 260117 already attached
05/01/2024 12:30:37;0008;pbs_mom;Job;373189.oracle;no active process for task 00000002
05/01/2024 12:30:37;0008;pbs_mom;Job;373189.oracle;no active process for task 00000002
05/01/2024 12:30:37;0080;pbs_mom;Job;373189.oracle;task 00000001 terminated
05/01/2024 12:30:37;0008;pbs_mom;Job;373189.oracle;Terminated
05/01/2024 12:30:37;0100;pbs_mom;Job;373189.oracle;task 00000001 cput=00:00:00
05/01/2024 12:30:37;0100;pbs_mom;Job;373189.oracle;task 00000002 cput=00:00:05
05/01/2024 12:30:37;0008;pbs_mom;Job;373189.oracle;kill_job
05/01/2024 12:30:37;0100;pbs_mom;Job;373189.oracle;oracle cput=00:00:05 mem=229672kb
05/01/2024 12:30:37;0100;pbs_mom;Job;373189.oracle;Obit sent
05/01/2024 12:30:37;0080;pbs_python;Hook;pbs_python;373189.oracle cleaned up successfully
05/01/2024 12:30:37;0080;pbs_mom;Job;373189.oracle;copy file request received
05/01/2024 12:30:37;0100;pbs_mom;Job;373189.oracle;Staged 2/2 items out over 0:00:00
05/01/2024 12:30:37;0008;pbs_mom;Job;373189.oracle;no active tasks
05/01/2024 12:30:37;0080;pbs_mom;Job;373189.oracle;delete job request received
05/01/2024 12:30:37;0008;pbs_mom;Job;373189.oracle;no active tasks
05/01/2024 12:30:38;0008;pbs_mom;Job;373189.oracle;kill_job"

正常执行日志 #

05/01/2024 22:55:27;0100;pbs_python;Hook;PBS_hpc_container;argv new: /opt/pbs/sbin/pbs_container 374116.oracle 0 /opt/pbs exec -w /home/seradmin 374116.oracle /bin/bash -c /var/spool/pbs/mom_priv/jobs/374116.oracle.SC
05/01/2024 22:55:27;0008;pbs_mom;Job;374116.oracle;Started, pid = 232270
05/01/2024 22:55:28;0008;pbs_mom;Job;374116.oracle;pid 233081 sid 233081 cmd 374116.orac attached as task 00000002
05/01/2024 22:55:28;0001;pbs_mom;Job;374116.oracle;tm_attach: sid 233081 already attached
05/01/2024 22:56:31;0008;pbs_mom;Job;374116.oracle;no active process for task 00000002
05/01/2024 22:56:31;0080;pbs_mom;Job;374116.oracle;task 00000001 terminated
05/01/2024 22:56:31;0008;pbs_mom;Job;374116.oracle;Terminated
05/01/2024 22:56:31;0100;pbs_mom;Job;374116.oracle;task 00000001 cput=00:00:01
05/01/2024 22:56:31;0100;pbs_mom;Job;374116.oracle;task 00000002 cput=00:01:50
05/01/2024 22:56:31;0008;pbs_mom;Job;374116.oracle;kill_job
05/01/2024 22:56:31;0100;pbs_mom;Job;374116.oracle;oracle cput=00:01:51 mem=3779196kb
05/01/2024 22:56:31;0100;pbs_mom;Job;374116.oracle;Obit sent
05/01/2024 22:56:32;0080;pbs_mom;Job;374116.oracle;copy file request received
05/01/2024 22:56:32;0100;pbs_mom;Job;374116.oracle;Staged 2/2 items out over 0:00:00
05/01/2024 22:56:32;0008;pbs_mom;Job;374116.oracle;no active tasks
05/01/2024 22:56:32;0080;pbs_mom;Job;374116.oracle;delete job request received
05/01/2024 22:56:41;0008;pbs_mom;Job;374116.oracle;no active tasks
05/01/2024 22:56:42;0008;pbs_mom;Job;374116.oracle;no active tasks
05/01/2024 22:56:42;0008;pbs_mom;Job;374116.oracle;no active tasks
05/01/2024 22:56:43;0008;pbs_mom;Job;374116.oracle;kill_job

解决方案 #

1. 任务启动前主动清理孤立容器 #

在PBS脚本开头添加容器清理逻辑，提前清除可能干扰当前任务的孤立容器，避免PBS钩子在任务执行中触发清理：

# 清理当前用户的已退出/无响应容器
docker rm -f $(docker ps -q -f status=exited -f status=dead 2>/dev/null) || true
# 清理与当前PBS任务ID关联的容器（如果容器命名规则包含任务ID）
docker rm -f $(docker ps -q -f name=${PBS_JOBID} 2>/dev/null) || true

2. 修改PBS容器钩子的清理策略 #

调整PBS的pbs_python钩子脚本，让孤立容器清理操作避开正在运行的任务：

• 检查容器对应的任务状态，仅清理已终止任务的孤立容器
• 延迟清理操作到任务执行完成后，而非任务运行期间

3. 增加容器状态检查与重试逻辑 #

在PBS脚本中，先验证当前任务对应的容器状态，确保容器正常后再执行Python脚本：

CONTAINER_NAME=${PBS_JOBID}
# 检查容器是否存在且正常运行
if ! docker inspect ${CONTAINER_NAME} >/dev/null 2>&1 || [ "$(docker inspect -f '{{.State.Running}}' ${CONTAINER_NAME})" != "true" ]; then
    echo "Container ${CONTAINER_NAME} is abnormal, reinitializing..."
    # 重新启动容器（需匹配PBS容器初始化的参数）
    docker run -d --name ${CONTAINER_NAME} --user seradmin -v /data:/data ${CONTAINER_IMAGE} sleep infinity
fi

4. 调整PBS任务的容器配置 #

在PBS脚本中添加容器自动清理参数，确保任务结束后自动清理容器，减少孤立容器产生：

#PBS -l container_cleanup=on_exit

同时检查PBS节点的mom_priv/config配置文件，确保container_default_cleanup=yes，全局开启容器退出后自动清理。

5. 优化Python脚本的鲁棒性 #

将Python任务拆分为多个阶段，通过状态文件记录执行进度，即使任务被中断，下次执行可从断点继续：

import os
STATUS_FILE = "/data/regional_model/src/forecast_progress.txt"

def run_data_fetch():
    # 数据获取阶段
    pass

def run_forecast():
    # 预测计算阶段
    pass

def run_validation():
    # 结果验证阶段
    pass

def main():
    current_stage = "data_fetch"
    if os.path.exists(STATUS_FILE):
        with open(STATUS_FILE, 'r') as f:
            current_stage = f.read().strip()
    
    try:
        if current_stage == "data_fetch":
            run_data_fetch()
            with open(STATUS_FILE, 'w') as f:
                f.write("forecast")
        if current_stage == "forecast":
            run_forecast()
            with open(STATUS_FILE, 'w') as f:
                f.write("validation")
        if current_stage == "validation":
            run_validation()
            os.remove(STATUS_FILE)
    except Exception as e:
        print(f"Stage {current_stage} failed: {str(e)}")
        # 保留状态文件，以便下次重试

if __name__ == "__main__":
    main()

内容的提问来源于stack exchange，提问作者Hrishikesh Satarkar