如何为Kafka Topic-1创建指定分区映射的3个消费者组

Great question! Let's break down how to assign specific partitions (0-1 to cg1, 2-3 to cg2, 4-5 to cg3) for your Topic-1. There are two main approaches depending on whether your consumer groups are new or already exist.

Method 1: Create New Consumer Groups with Manual Partition Assignment #

If your consumer groups (cg1, cg2, cg3) don't exist yet, you can start console consumers with explicit partition assignments. This registers the partition mapping directly with each group as you launch the consumers.

Run each of these commands in separate terminals (replace <your-bootstrap-server> with your actual Kafka bootstrap address, e.g., localhost:9092):

For consumer group cg1 (partitions 0 and 1): #

kafka-console-consumer.sh --bootstrap-server <your-bootstrap-server> \
  --consumer-property group.id=cg1 \
  --assign '{"topic":"Topic-1","partitions":[0,1]}'

For consumer group cg2 (partitions 2 and 3): #

kafka-console-consumer.sh --bootstrap-server <your-bootstrap-server> \
  --consumer-property group.id=cg2 \
  --assign '{"topic":"Topic-1","partitions":[2,3]}'

For consumer group cg3 (partitions 4 and 5): #

kafka-console-consumer.sh --bootstrap-server <your-bootstrap-server> \
  --consumer-property group.id=cg3 \
  --assign '{"topic":"Topic-1","partitions":[4,5]}'

Key note: Using --assign tells Kafka to use manual partition assignment instead of the default group-based assignment strategy. This ensures each group will only consume the specified partitions.

Method 2: Reassign Partitions for Existing Consumer Groups #

If the consumer groups already exist (and have existing partition assignments), you can use the kafka-consumer-groups.sh tool to reconfigure their partition mappings.

Step 1: Create a reassignment JSON file #

Create a file named partition-reassignment.json with the following content:

{
  "version": 1,
  "groups": [
    {
      "group": "cg1",
      "topicPartitions": {
        "Topic-1": [0, 1]
      }
    },
    {
      "group": "cg2",
      "topicPartitions": {
        "Topic-1": [2, 3]
      }
    },
    {
      "group": "cg3",
      "topicPartitions": {
        "Topic-1": [4, 5]
      }
    }
  ]
}

Step 2: Execute the reassignment #

First, stop any running consumers in the target groups to avoid conflicts. Then run this command:

kafka-consumer-groups.sh --bootstrap-server <your-bootstrap-server> \
  --reassign-groups \
  --reassignment-json-file partition-reassignment.json

Step 3: Verify the assignment #

To confirm the partitions were assigned correctly, run:

kafka-consumer-groups.sh --bootstrap-server <your-bootstrap-server> \
  --describe --group cg1

Repeat this for cg2 and cg3 to check their partition mappings.

Important Notes #

• Manual partition assignment overrides Kafka's default partition assignment strategies (like Range or RoundRobin). Once assigned, the group will stick to these partitions unless you reconfigure it.
• If you're using consumer applications outside the console, you can implement manual assignment in your code using the assign() method instead of subscribe().

内容的提问来源于stack exchange，提问作者hariram manohar