如何迁移Kafka至其他服务器？如何备份元信息部署新Kafka服务器？

Great question—let's break this down step by step, since migrating Kafka servers and securing metadata backups are critical tasks to avoid data loss or downtime.

First, let's cover moving your existing Kafka setup to a new server. The process varies a bit for single-node vs. clustered Kafka, but the core steps are:

1. Migrate Kafka Data #

Kafka stores all topic data in the directory defined by log.dirs in your server.properties (default is /tmp/kafka-logs). Here's how to move it:

• Stop the old Kafka broker first (and Zookeeper if you're using it, or KRaft controller for newer versions) to avoid writing new data during migration.
• Copy the entire log.dirs directory to the new server using a tool like scp or rsync for efficiency:

  rsync -avz /path/to/old/kafka-logs user@new-server:/path/to/new/kafka-logs
  

• Ensure the new directory has the same permissions as the old one (match the user/group running Kafka, usually kafka).

2. Migrate Configuration Files #

• Copy the core configuration files from the old server to the new one:
  • server.properties (main broker config)
  • consumer.properties (default consumer config, if used)
  • producer.properties (default producer config, if used)
  • For Zookeeper setups: zookeeper.properties
  • For KRaft setups: kraft/server.properties and the cluster ID file
• Update any server-specific values in the new configs, like listeners, advertised.listeners, broker.id, and log.dirs (if you changed the path on the new server).

Before shutting down your old Kafka instance, you’ll want to capture all metadata to replicate the exact state on the new server. Use Kafka’s built-in command-line tools (found in the bin/ directory of your Kafka installation) to pull this info:

1. Topic & Partition Metadata #

• List all topics:

  kafka-topics.sh --list --bootstrap-server old-kafka-ip:9092
  

• Get detailed info for all topics (including partitions, replicas, ISR, retention settings):

  kafka-topics.sh --describe --bootstrap-server old-kafka-ip:9092 > kafka-topics-metadata.txt
  

2. Consumer Group Metadata #

• List all consumer groups:

  kafka-consumer-groups.sh --list --bootstrap-server old-kafka-ip:9092
  

• Get detailed offset info for all groups (to replicate consumption state):

  kafka-consumer-groups.sh --describe --all-groups --bootstrap-server old-kafka-ip:9092 > kafka-consumer-groups-metadata.txt
  

3. Broker & Topic-Level Configs #

• Get custom broker configurations (if any):

  kafka-configs.sh --describe --entity-type brokers --entity-name <old-broker-id> --bootstrap-server old-kafka-ip:9092 > kafka-broker-configs.txt
  

• Get custom topic-level configurations (overrides from default):

  kafka-configs.sh --describe --entity-type topics --bootstrap-server old-kafka-ip:9092 > kafka-topic-configs.txt
  

Once you have the metadata and migrated data/configs, set up the new server to match the old one:

1. Install Kafka on the new server (same version as the old one to avoid compatibility issues).
2. Paste the migrated config files and update server-specific values (like broker.id, listeners).
3. Verify the data directory is in place and has correct permissions.
4. Recreate topics (if needed) – if you didn’t migrate the log.dirs directory (e.g., starting fresh), use the kafka-topics-metadata.txt to create each topic with the same partitions, replicas, and configs:

   kafka-topics.sh --create --topic my-topic --partitions 3 --replication-factor 1 --bootstrap-server new-kafka-ip:9092 --config retention.ms=86400000
   

5. Restore consumer offsets (if migrating consumption state): Use the kafka-consumer-groups-metadata.txt to reset offsets on the new broker, or use the kafka-consumer-groups.sh tool to set offsets manually.
6. Start the new Kafka broker (and Zookeeper/KRaft controller if applicable) and verify all topics, partitions, and consumer groups are present.

Yes, there are established backup strategies for Kafka:

• Data Backup:
  • Use filesystem snapshots (e.g., LVM snapshots, AWS EBS snapshots) on the log.dirs directory. Since Kafka writes logs sequentially, snapshots capture a consistent state as long as the broker is paused or stopped.
  • For clustered setups, leverage replication: Ensure topics have enough replicas (minimum 3) so data is redundant across brokers.
• Metadata Backup:
  • Script the metadata capture commands above to run periodically (e.g., via cron) and save the output files to a secure location (like a cloud storage bucket or backup server).
  • For Zookeeper setups, back up the Zookeeper data directory (dataDir in zookeeper.properties), as it stores cluster metadata. For KRaft, back up the metadata log directory.
• Live Migration:
  • Use MirrorMaker 2 (built into Kafka) to replicate data from the old cluster to the new one in real-time. This allows you to switch producers/consumers to the new cluster with minimal downtime, without needing to shut down the old broker first.

Key Note: Always test the migration/backup process in a staging environment first to catch issues before affecting production.

内容的提问来源于stack exchange，提问作者T T