Node应用中Kue.js替换AWS SQS的并发处理疑问

Hey there! Since you're making the switch from AWS SQS to Kue.js, let's break down your questions about task processing behavior—this stuff is key to getting your queue setup running smoothly.

1. Concurrent Task Processing Count #

By default, Kue processes tasks one at a time for a given task type. But you can easily adjust the concurrency level when calling queue.process() to handle multiple tasks in parallel.

Here's how you set it:

// Process up to 5 "notification" tasks simultaneously
queue.process('notification', 5, (job, done) => {
  // Your task logic goes here
  sendPushNotification(job.data, done);
});

The second parameter (5 in this example) defines how many concurrent tasks this worker instance will handle. If you skip this parameter, it defaults to 1 (sequential processing).

2. Sequential vs. Batch Processing #

Kue doesn't do "batch processing" in the sense of fetching multiple tasks at once to process as a group. Instead, it uses parallel task execution based on your concurrency setting. Each task gets its own separate invocation of your handler function, running concurrently (powered by Node.js's event loop for async operations).

So with a concurrency of 4, you'll have 4 independent task handlers running side-by-side, each handling one individual task—no batching of task data into a single handler call.

3. Multiple queue.process() Calls for the Same Task Type #

If you call queue.process() for the same task type in multiple places (whether in the same Node process or across separate processes), here's what to expect:

• Each call registers a new set of workers (with their own concurrency count) for that task type.
• Kue will distribute incoming tasks across all available workers (from every registered process() call) in a load-balanced way.
• If you register different handler logics for the same task type, tasks will be randomly assigned to any of the registered handlers—so be intentional with this! It’s best to keep task type handlers consistent unless you specifically want multiple processing paths.

Example of multiple process calls in the same application:

// Module 1: Handles "invoice" tasks with concurrency 2
queue.process('invoice', 2, (job, done) => {
  generatePDFInvoice(job.data, done);
});

// Module 2: Also handles "invoice" tasks with concurrency 3
queue.process('invoice', 3, (job, done) => {
  sendInvoiceEmail(job.data, done);
});

In this scenario, you’ll have 5 total concurrent workers for "invoice" tasks, and each task could be picked up by either of the two handlers.

内容的提问来源于stack exchange，提问作者Anirudh