BullMQ队列部分任务未执行即退出问题排查求助

背景 #

通过Puppeteer生成PDF证书，再根据收件人信息发送邮件，使用Worker处理队列任务。目前所有任务已入队，但部分任务未执行任何核心操作就提前退出。

Worker逻辑代码 #

const sendCertificatesWorker = new Worker(queueName, async job => {
  const {
    data: { id, name, email, link },
    collectionId,
    nameOfOrganization,
    thumbnail,
    raw
  } = job.data;

  // Generate Certificate
  const filePath = await GenerateCertificatePdfs(job.data.data, raw);
  await job.updateProgress("certificate generated");

  // Send email
  const result = await sendCertificatesEmail({
    name,
    email,
    link,
    filePath,
    nameOfOrganization,
    thumbnail
  });

  if (result.error) {
    await job.updateProgress(
      `email not sent and the reason is: ${result.message}`
    );
    await Certificate.updateOne(
      { _id: id },
      { $set: { status: "not sent" } }
    );
    await job.updateProgress(
      `cert with id ${id} updated in database as not sent`
    );
  } else {
    await job.updateProgress(`email sent`);
    await Certificate.updateOne({ _id: id }, { $set: { status: "sent" } });
    await job.updateProgress(
      `cert with id ${id} updated in database as sent`
    );
  }

  // Update collection
  const collection = await Collection.findById(collectionId);
  if (collection) {
    await job.updateProgress("updating the collection..");
    await collection.save();
  }

  return { sent: result.error };
}, { connection: workerConnection });

问题现象（服务器日志） #

A job with ID 22 is waiting
A job with ID 23 is waiting
A job with ID 24 is waiting
Job 22 is now active; previous status was waiting
Job 23 is now active; previous status was waiting
job with 23 is currently in progress: certificate generated
job with 23 is currently in progress: email sent
job with 23 is currently in progress: cert with id 64afda0da236e94d7290bd47 updated in database as sent
job with 23 is currently in progress: updating the collection..
23 has completed and returned false
Job 24 is now active; previous status was waiting
23 has completed!
22 has completed and returned false
job with 24 is currently in progress: certificate generated
job with 24 is currently in progress: email sent
job with 24 is currently in progress: cert with id 64afda0da236e94d7290bd3f updated in database as sent
job with 24 is currently in progress: updating the collection..
24 has completed and returned false
24 has completed!

Job Id 22 exited without generating the pdf nor email it.

已尝试的Redis配置 #

// Configure redis connection
const queueConnection = new IORedis({
  host: "localhost", // Redis server hostname
  port: 6379, // Redis server port
  maxmemoryPolicy: "noeviction",
  maxRetriesPerRequest: null,
  enableOfflineQueue: false,
  retryStrategy() {
    const delay = Math.min(3 * 50, 2000);
    return delay;
  }
});

const workerConnection = new IORedis({
  host: "localhost", // Redis server hostname
  port: 6379, // Redis server port
  maxmemoryPolicy: "noeviction",
  maxRetriesPerRequest: null,
  enableOfflineQueue: true,
  retryStrategy() {
    const delay = Math.min(3 * 50, 2000);
    return delay;
  }
});

排查建议 #

• 捕获任务中的所有未处理异常：当前Worker逻辑没有全局错误捕获，若GenerateCertificatePdfs调用、数据解构或其他异步操作抛出未捕获异常，会直接导致任务崩溃退出。在Worker的async函数外层添加try-catch块，捕获错误后记录详细日志，更新任务进度与数据库状态：

  async job => {
    try {
      // 原任务逻辑
    } catch (err) {
      await job.updateProgress(`任务失败：${err.message}`);
      await Certificate.updateOne({ _id: job.data.data.id }, { $set: { status: "failed" } });
      console.error(`任务${job.id}失败详情：`, err.stack);
      throw err; // 让队列记录失败状态
    }
  }
  

• 检查异常任务的数据合法性：获取ID22任务的完整数据，确认job.data中的id、raw、data等字段是否存在或格式正确。可在任务开始时打印job.data，对比正常任务（23、24）的数据差异。
• 限制Worker并发数与优化Puppeteer资源占用：Puppeteer启动Chromium实例会消耗大量内存，若Worker并发数过高，可能导致系统因内存不足强制终止任务。将Worker的concurrency设置为1或2，同时给Puppeteer添加启动参数：

  // GenerateCertificatePdfs中启动浏览器的配置
  const browser = await puppeteer.launch({
    headless: "new",
    args: [
      "--disable-dev-shm-usage",
      "--no-sandbox",
      "--disable-gpu",
      "--single-process"
    ]
  });
  

• 检查Redis连接与队列状态：监听Worker的Redis连接错误事件，记录连接异常；使用队列的getJob(22)方法获取任务的失败原因、堆栈信息，确认是否因Redis断开导致任务中断。
• 单独验证GenerateCertificatePdfs函数：传入ID22任务的job.data.data和raw参数，单独调用该函数，检查是否会抛出异常、卡住或无法生成文件（比如文件路径权限不足、模板加载失败）。
• 检查进程退出信号：在服务器上监控进程退出状态，若任务进程被系统杀死，可通过dmesg或系统日志查看是否有OOM（内存不足）记录。

内容的提问来源于stack exchange，提问作者Olanitori Tobi