寻求支持Queue of Queues的Queue Broker：解决重复人员数据消息问题

Hey there, let's break this down for you. Your scenario isn't actually that unusual—dealing with duplicate messages at scale is a common pain point, and the "queue of queues" pattern can be a solid approach here, but you might not need a specialized broker out of the box. Let's walk through solutions and recommendations:

Don't second-guess your approach. Duplicate user demographic messages at the million-scale level pop up in tons of use cases (like user profile syncs, CRM data ingestion, etc.). The core needs here are:

1. Deduplicate messages tied to the same unique user ID
2. Route/process these messages in a way that avoids overwhelming consumers (the queue-of-queues pattern helps with this)

You don't need a niche broker—tools you probably already know can handle this with a bit of configuration:

Step 1: Pre-Queue Deduplication #

Add a lightweight deduplication layer before messages hit your broker to filter out most duplicates upfront:

• Use a distributed cache like Redis: For each incoming message, check if the user's unique ID exists in a Redis Set with SISMEMBER user_processed {user_id}. If it doesn't exist, add it with SADD user_processed {user_id} and send the message to the broker.
• Pro tip: Set a TTL on these keys (e.g., 24 hours) or periodically clean up IDs for users whose messages have been fully processed to avoid memory bloat.

Step 2: Simulate Queue-of-Queues with Routing/Partitioning #

For RabbitMQ #

Use a Topic Exchange with a routing key derived from the user ID (e.g., hash the user ID and take modulo 1000 to get a key like user.001). Bind 1000+ child queues to this exchange, each listening to a specific routing key. This ensures:

• All messages for the same user go to the same child queue
• You can scale consumers independently per child queue
• Consumers can dedupe any remaining duplicates locally (e.g., with a Guava Cache)

For Kafka #

Implement a custom partitioner that maps each user ID to a fixed partition. Each Kafka partition acts as a "child queue"—since partitions are processed sequentially by consumers in a group, you get:

• Ordered processing for the same user's messages
• Built-in scalability via consumer groups handling different partitions
• Deduplication can happen in the consumer using a local cache or database unique constraint

If you want a tool that supports this pattern out of the box, these options fit the bill:

• NATS JetStream: JetStream's streams and consumer filters let you create per-user or grouped child streams. You can route messages to specific streams based on user ID, and manage the lifecycle of these streams via the main broker. It's built for high throughput, making it perfect for million-scale traffic.
• Apache Pulsar: Pulsar's partitioned topics and namespaces enable a natural queue-of-queues setup. You can create a namespace for user data, then either use partitioned topics (grouped by user ID hash) or even per-user topics (though hash grouping is better to avoid resource bloat). Pulsar's layered storage also handles large volumes without performance hits.
• Celery (with Redis/RabbitMQ): While it's a task queue, Celery lets you create dedicated task queues per user group (or even per user) and manage them via a main queue. It's great if you need to tie message processing directly to execution logic, rather than just routing messages.

• Don't create a queue per user: With millions of users, this will blow up your broker's resource usage. Stick to hash-based grouping (e.g., 1000-10,000 child queues) to balance order guarantee and resource efficiency.
• Don't skip post-consumer deduplication: Network delays or race conditions can slip duplicates past the pre-queue layer. Always validate uniqueness at the consumer (e.g., using a database unique constraint on the user ID + message timestamp).

内容的提问来源于stack exchange，提问作者AHungerArtist