如何将Kafka主题分区分配到指定的多个Kafka服务器？

Got it, let's walk through exactly how to assign your Test topic's 4 partitions to brokers k1 and k2 as you specified (k1 hosts partitions 0 & 1, k2 hosts partitions 2 & 3). First, a quick key detail: Kafka uses broker IDs (not hostnames like k1/k2) to handle partition assignments, so confirm the broker.id value in each broker's server.properties first. For this example, we'll assume k1 has broker.id=1 and k2 has broker.id=2—swap these with your actual IDs if needed.

Option 1: Assign Partitions When Creating the Topic #

If the "Test" topic doesn't exist yet, you can define the partition assignment directly during creation using the kafka-topics.sh tool:

kafka-topics.sh --create \
  --bootstrap-server k1:9092,k2:9092 \
  --topic Test \
  --partitions 4 \
  --replica-assignment "1,1,2,2"

Breakdown of the command: #

• --replica-assignment: Comma-separated list where each entry maps a partition to its broker(s). Since you want single replicas (each partition lives on one broker), we list 1 for partitions 0 and 1 (k1), then 2 for partitions 2 and 3 (k2). If you wanted multi-replica partitions, you'd format each entry like 1,2 (e.g., "1,2,1,2" for 2 replicas per partition).

Option 2: Reassign Partitions for an Existing Topic #

If the "Test" topic already exists and you need to rebalance its partitions to match your desired setup, follow these steps:

1. Create a reassignment JSON file (e.g., partition-assignment.json) with your target partition mapping:

{
  "version": 1,
  "partitions": [
    {"topic": "Test", "partition": 0, "replicas": [1]},
    {"topic": "Test", "partition": 1, "replicas": [1]},
    {"topic": "Test", "partition": 2, "replicas": [2]},
    {"topic": "Test", "partition": 3, "replicas": [2]}
  ]
}

2. Execute the reassignment using kafka-reassign-partitions.sh:

kafka-reassign-partitions.sh --bootstrap-server k1:9092,k2:9092 \
  --reassignment-json-file partition-assignment.json \
  --execute

3. Verify the assignment was successful:

kafka-reassign-partitions.sh --bootstrap-server k1:9092,k2:9092 \
  --reassignment-json-file partition-assignment.json \
  --verify

Important Notes #

• Replace k1:9092,k2:9092 with your actual Kafka broker addresses and port numbers.
• Partition reassignment triggers data migration between brokers—avoid running this during peak traffic to prevent performance impacts.
• Double-check your broker IDs: Mismatched IDs will lead to failed assignments.

内容的提问来源于stack exchange，提问作者biunovich