GCE上基于Firebase-Queue的应用优雅停机方案设计问询

Hey there! Let’s walk through a solid design for handling long-running Firebase Queue tasks when your GCE workers get terminated (whether from auto-scaling down or manual shutdown). The goal here is to make sure in-progress tasks finish before the VM goes down, while keeping your auto-scaling workflow intact. Here’s how to do it:

GCE sends a SIGTERM signal to instances 30 seconds (by default) before forcing termination with SIGKILL. You can extend this grace period up to 3600 seconds (1 hour) — critical for your long-running tasks.

In your worker code, add a listener for SIGTERM that triggers an orderly shutdown:

// Example for Node.js Firebase Queue worker
let isShuttingDown = false;
let currentTask = null;

process.on('SIGTERM', async () => {
  console.log('Received termination signal — starting graceful shutdown');
  isShuttingDown = true;

  // Stop pulling new tasks from Firebase Queue
  await queue.shutdown(); // Use your Firebase Queue client's shutdown method

  // Wait for current task to finish if one is in progress
  if (currentTask) {
    await finishCurrentTask(currentTask);
    updateTaskStatus(currentTask, 'completed');
  }

  process.exit(0);
});

// Modify task processing loop to track current task
queue.process(async (task) => {
  if (isShuttingDown) return;
  currentTask = task;
  updateTaskStatus(task, 'in_progress');
  await handleLongRunningTask(task);
  currentTask = null;
});

Tweak your Firebase Queue settings to support in-progress task completion:

• Set lockTimeout to a value longer than your maximum expected task duration. This prevents other workers from picking up your in-progress task while you’re finishing it.
• Add a custom task status field (like status: "in_progress") that your worker updates when starting a task. This makes it easy to track which tasks are actively being handled if a shutdown happens mid-process.
• Avoid short snapshot polling intervals during shutdown — your worker should focus on finishing the current task, not polling for new ones.

By default, GCE only gives 30 seconds for shutdown. For long tasks, you need to increase this in your instance template:

• Go to your GCE Instance Template settings
• Under "Management", add a metadata entry with key shutdown-script (you can use a simple script to log shutdown events, or leave it empty — the grace period is controlled by the instance group's timeout)
• When configuring your auto-scaling instance group, set the "Termination grace period" to match your longest task runtime (e.g., 1200 seconds for 20-minute tasks)

This ensures your worker has enough time to wrap up the current task before the VM is killed.

• Stop pulling new tasks immediately when isShuttingDown is true. Modify your task processing loop to check this flag before fetching the next task.
• Make tasks idempotent: If a task does get interrupted (e.g., grace period runs out), ensure re-running it won’t cause duplicate work or data issues. Add unique task IDs and check for existing completion records before processing.
• Track task progress: For extra safety, log task progress to Firebase (e.g., progress: 75%) so if a shutdown forces an early exit, the next worker can resume from where it left off instead of starting over.

Set up monitoring to catch issues before they become problems:

• Track instances terminated by SIGKILL (this means your grace period is too short) using Stackdriver.
• Monitor Firebase task statuses for unexpected interrupted or failed states.
• Alert on high rates of task interruptions — this is a sign you need to adjust your grace period or optimize task runtime.

内容的提问来源于stack exchange，提问作者Jamieson Bates