First off: Yes, this worker recycling approach is absolutely applicable to your 4-logical-core cluster setup—but it’s important to frame it correctly: this is a temporary mitigation for memory leaks, not a permanent fix. Let’s break this down.

Why It Works for Your Architecture #

Your setup uses 4 independent worker processes (one per logical core), which means you can restart them one at a time without taking down the entire service. Each worker is an isolated OS process, so destroying and spawning a new one will fully free up any memory that leaked in the old worker’s heap or native resources. This is a common stopgap when dealing with hard-to-track leaks (e.g., from third-party libraries, persistent event listeners, or unclosed handles) that take weeks to accumulate.

That said: don’t rely on this as your only solution. You should still prioritize finding and fixing the root cause of the leak (tools like clinic.js, heapdump, or Node’s built-in --inspect flag can help with this). Worker recycling is just a band-aid to keep your service stable while you diagnose the real issue.

How to Implement Safe Worker Recycling (Step-by-Step) #

The key here is to avoid service downtime by restarting workers sequentially, ensuring graceful shutdowns, and verifying new workers are ready before routing traffic to them. Here’s how to do it:

1. Define a Trigger for Recycling #

Choose when to recycle workers based on either:

• Memory thresholds: Monitor each worker’s memory usage (use process.memoryUsage().heapUsed or rss in the worker, or have workers report metrics to the master). For example, trigger a restart when heapUsed exceeds 500MB.
• Time intervals: Schedule weekly/daily restarts during low-traffic windows (if your traffic has predictable lulls).

2. Sequential Worker Restart Logic in the Master Process #

Never restart all workers at once. The master should handle one worker at a time:

const cluster = require('cluster');
const numCPUs = require('os').cpus().length;

// Function to safely restart a single worker
async function restartWorker(worker) {
  // Step 1: Stop sending new requests to the worker
  worker.disconnect();

  // Step 2: Wait for the worker to finish existing requests or timeout after 30s
  await new Promise((resolve) => {
    worker.on('exit', resolve);
    setTimeout(resolve, 30000); // Fallback if worker hangs
  });

  // Step 3: Spawn a new worker
  const newWorker = cluster.fork();
  // Wait for new worker to be ready
  await new Promise((resolve) => {
    newWorker.on('message', (msg) => {
      if (msg.type === 'ready') resolve();
    });
  });
}

// Example: Restart workers one by one when a threshold is hit
setInterval(() => {
  const workers = Object.values(cluster.workers);
  for (const worker of workers) {
    if (worker.memoryUsage.heapUsed > 500 * 1024 * 1024) { // 500MB
      restartWorker(worker);
      break; // Restart one at a time
    }
  }
}, 60000); // Check every minute

3. Graceful Shutdown in the Worker Process #

Your Koa worker needs to handle shutdown signals properly to avoid dropping in-flight requests:

const Koa = require('koa');
const app = new Koa();

// Track active requests
let activeRequests = 0;

app.use(async (ctx, next) => {
  activeRequests++;
  try {
    await next();
  } finally {
    activeRequests--;
    // If no active requests and shutdown is pending, exit
    if (shutdownPending && activeRequests === 0) {
      process.exit(0);
    }
  }
});

// Start server
const server = app.listen(3000, () => {
  // Notify master that worker is ready
  process.send({ type: 'ready' });
});

let shutdownPending = false;

// Handle termination signals
process.on('SIGTERM', () => {
  shutdownPending = true;
  // Stop accepting new connections
  server.close(() => {
    // If no active requests, exit immediately
    if (activeRequests === 0) process.exit(0);
  });
});

4. Add Monitoring & Alerts #

• Track worker restart counts, memory usage trends, and request success rates.
• Set up alerts for:
  • Frequent worker restarts (e.g., more than 2 per hour)
  • Memory usage spiking faster than usual
  • Failed worker spawns

5. Test the Workflow #

Test the recycling process in a staging environment first:

• Simulate high traffic to ensure in-flight requests aren’t dropped during restarts.
• Verify that new workers initialize correctly (connect to databases, load configs, etc.) before handling traffic.

内容的提问来源于stack exchange，提问作者Gatmando