Azure Event Hub负载分配咨询：多订阅者并行处理事件需求

Great question! Let’s walk through how to build out your desired Azure Event Hub setup—parallel subscribers targeting specific event types, plus distributed compute for processing tasks. Here’s a step-by-step breakdown tailored to your scenario:

Right now, without partitions, you’re limited to one active receiver per consumer group—that’s why you can’t scale out parallel processing. Partitions are the core of Event Hub’s scalability: each partition acts as an independent data stream, allowing multiple receivers to consume events in parallel.

• When updating your Event Hub, choose a partition count that aligns with your maximum expected parallel processing needs (note: partition count can’t be changed after creation, so err on the side of more if you anticipate growth). For example, if you want 5 parallel streams for different event types, start with at least 5 partitions.

Consumer groups let multiple sets of subscribers read the entire Event Hub stream independently—perfect for splitting EventA and EventB into separate processing pipelines.

• Create a unique consumer group for each subscriber type: e.g., cg-event-a-handler for EventA receivers, cg-event-b-handler for EventB receivers.
• Each consumer group maintains its own checkpoint (last processed event position), so one subscriber’s progress won’t affect others.

You have two solid options here, depending on how complex your routing needs are:

Option A: Filter at the Subscriber Level (Simple & Direct) #

For basic type-based routing, have each subscriber receive all events from their consumer group, then filter for their target event type in code. This is easy to implement and works well if your event payloads include a clear identifier (like an eventType property).

Example C# snippet for an EventA subscriber:

await using var consumer = new EventHubConsumerClient(
    "cg-event-a-handler", 
    "<your-event-hub-connection-string>", 
    "<your-event-hub-name>"
);

await foreach (var partitionEvent in consumer.ReadEventsAsync(CancellationToken.None))
{
    var eventData = partitionEvent.Data;
    
    // Extract event type from properties or payload
    if (eventData.Properties.TryGetValue("eventType", out var eventTypeObj) && 
        eventTypeObj.ToString() == "EventA")
    {
        // Run your compute task for EventA
        await ProcessEventA(eventData.Body.ToString());
        
        // Update checkpoint to mark this event as processed
        await consumer.UpdateCheckpointAsync(partitionEvent);
    }
}

Option B: Use Event Grid for Advanced Routing (No Filtering Code Needed) #

If you want to avoid having subscribers receive all events, integrate Event Hub with Azure Event Grid. Event Grid can route events directly to specific compute targets based on event properties (like eventType), so only the intended subscriber gets the event.

• Configure Event Grid to subscribe to your Event Hub, then set up filter rules (e.g., "send all events where eventType = 'EventA' to Function App A").
• Targets can include Azure Functions, VMs, AKS pods, or even other Event Hubs—great for distributing compute across different services.

Once events are routed to the right subscribers, here are the best ways to handle distributed compute:

• Azure Functions: Use Event Hub-triggered Functions tied to each consumer group. Functions auto-scale based on event load, so you don’t have to manage infrastructure. Each Function can run your compute logic for its target event type.
• Azure Kubernetes Service (AKS): Deploy pods as event consumers—each pod handles one event type. AKS lets you scale pods horizontally based on queue length, and you can use Kubernetes Jobs for long-running compute tasks if needed.
• Azure Virtual Machine Scale Sets (VMSS): If you need a custom environment, deploy VM instances as subscribers. VMSS auto-scales based on metrics like CPU usage or event backlog, ensuring you have enough capacity for compute tasks.

• Checkpointing: Always implement checkpointing in your subscribers to avoid reprocessing events after restarts. Most SDKs and triggers (like Functions) handle this automatically, but double-check for custom implementations.
• Scaling Limits: For a single consumer group, you can’t have more active receivers than partitions. If you need more parallelism than your partition count allows, add more consumer groups (each can have up to N receivers, where N is partition count).
• Event Payload Consistency: Make sure all events include a consistent identifier (like eventType) so filtering/routing works reliably.

内容的提问来源于stack exchange，提问作者Kayani