PM2集群模式下Express优雅关机事件处理器无法完整执行

问题描述 #

Express应用实现了优雅关机函数，直接运行时正常工作；但通过PM2以集群模式启动时，函数仅执行第一行console.log('test')就终止，后续清理逻辑未执行。日志中未发现SIGKILL信号，调整PM2的kill timeout、使用no-killtree参数均无效。

PM2日志片段 #

2022-11-09T15:23:28: PM2 log: App [server:7] starting in -cluster mode-
2022-11-09T15:23:28: PM2 log: App [server:7] online
2022-11-09T15:23:28: PM2 log: App [server:8] starting in -cluster mode-
2022-11-09T15:23:28: PM2 log: App [server:8] online
2022-11-09T15:23:28: PM2 log: App [server:9] starting in -cluster mode-
2022-11-09T15:23:28: PM2 log: App [server:9] online
2022-11-09T15:23:28: PM2 log: App [server:10] starting in -cluster mode-
2022-11-09T15:23:28: PM2 log: App [server:10] online
2022-11-09T15:39:26: PM2 log: Stopping app:server id:7
2022-11-09T15:39:26: PM2 log: Stopping app:server id:8
2022-11-09T15:39:27: PM2 log: App name:server id:8 disconnected
2022-11-09T15:39:27: PM2 log: App [server:8] exited with code [0] via signal [SIGINT]
2022-11-09T15:39:27: PM2 log: App name:server id:7 disconnected
2022-11-09T15:39:27: PM2 log: App [server:7] exited with code [0] via signal [SIGINT]
2022-11-09T15:39:27: PM2 log: pid=50526 msg=process killed
2022-11-09T15:39:27: PM2 log: pid=50533 msg=process killed
2022-11-09T15:39:27: PM2 log: Stopping app:server id:9
2022-11-09T15:39:27: PM2 log: Stopping app:server id:10
2022-11-09T15:39:27: PM2 log: App name:server id:9 disconnected
2022-11-09T15:39:27: PM2 log: App [server:9] exited with code [0] via signal [SIGINT]
2022-11-09T15:39:27: PM2 log: App name:server id:10 disconnected
2022-11-09T15:39:27: PM2 log: App [server:10] exited with code [0] via signal [SIGINT]
2022-11-09T15:39:27: PM2 log: pid=50540 msg=process killed
2022-11-09T15:39:27: PM2 log: pid=50547 msg=process killed

事件处理器代码 #

const onSignal = (signal) => {
        console.log('test');
        console.log(`${signal} signal received. Cleaning up and shutting down.`);
        console.log('test');

        console.log("Closing HTTPS server to new connections");

        httpsServer.close(error => {
                if (error) {
                        console.log(`Error closing HTTPS server: ${error.message}`);
                        console.log("Process cleanup did not occur due to error.");
                        process.exit(1);
                }
        });

        console.log("Closing SQL Connection Pool");

        db.end().then(() => {
                console.log("SQL Connection Pool successfully ended");
        })
        .catch(reason => {
                console.error(`Failed to close SQL Connection Pool. Error: ${reason.message}`);
        })
        .finally(() => {
                process.exit(0);
        });
};

process.on('SIGINT', onSignal);
process.on('SIGQUIT', onSignal);
process.on('SIGTERM', onSignal);

解决方案 #

1. 修复异步流程，等待所有清理操作完成 #

原代码并行执行服务器关闭和数据库关闭，且提前调用process.exit导致清理逻辑未完成就终止。改为先等待服务器关闭所有连接，再关闭数据库，最后退出：

const onSignal = async (signal) => {
  console.log('test');
  console.log(`${signal} signal received. Cleaning up and shutting down.`);
  console.log('test');

  // 强制刷新日志缓冲，避免内容未输出就退出
  process.stdout.flush();

  console.log("Closing HTTPS server to new connections");
  process.stdout.flush();

  // 等待服务器关闭所有现有连接
  await new Promise((resolve, reject) => {
    httpsServer.close(error => {
      if (error) {
        console.log(`Error closing HTTPS server: ${error.message}`);
        console.log("Process cleanup did not occur due to error.");
        reject(error);
      } else {
        console.log("HTTPS server closed successfully");
        resolve();
      }
    });
  });

  console.log("Closing SQL Connection Pool");
  process.stdout.flush();

  // 等待数据库连接池关闭
  try {
    await db.end();
    console.log("SQL Connection Pool successfully ended");
  } catch (reason) {
    console.error(`Failed to close SQL Connection Pool. Error: ${reason.message}`);
  } finally {
    process.exit(0);
  }
};

process.on('SIGINT', onSignal);
process.on('SIGQUIT', onSignal);
process.on('SIGTERM', onSignal);

2. 配置PM2集群模式优雅关机参数 #

在ecosystem.config.js中添加以下配置，确保PM2等待进程完成清理：

module.exports = {
  apps: [{
    name: 'server',
    script: './app.js',
    instances: 'max',
    exec_mode: 'cluster',
    // 延长PM2等待进程退出的超时时间（单位：毫秒）
    kill_timeout: 60000,
    // 启用就绪通知，让应用启动时告知PM2已就绪
    wait_ready: true,
    // 配置日志输出，避免缓冲
    out_file: './logs/out.log',
    error_file: './logs/error.log',
    merge_logs: true,
    log_date_format: 'YYYY-MM-DD HH:mm:ss'
  }]
};

同时在应用启动成功后添加就绪通知：

httpsServer.listen(port, () => {
  console.log(`Server running on port ${port}`);
  // 告知PM2应用已就绪
  if (process.send) {
    process.send('ready');
  }
});

3. 主动通知PM2关机状态 #

在信号处理函数开头添加代码，阻止PM2提前终止进程：

const onSignal = async (signal) => {
  // 告知PM2进程正在执行关机清理
  if (process.send) {
    process.send('shutdown');
  }

  // 后续清理逻辑...
};

关键原因分析 #

1. 异步流程未正确等待：原代码并行执行清理操作且提前退出，导致逻辑未完成就终止进程。
2. 日志缓冲问题：Node.js的console.log存在缓冲，进程快速退出时后续日志未及时输出，造成仅执行第一行的假象。
3. PM2集群IPC机制：PM2依赖IPC连接判断进程状态，若进程未主动通知关机状态，PM2可能提前终止进程。

内容的提问来源于stack exchange，提问作者pdmoerman