MirrorMaker2仅复制约1/3数据的故障排查与解决请求

Hey there, let's break down what's happening with your MirrorMaker2 setup and fix those issues one by one.

First: Why the ACL Error Doesn't Block Partial Replication #

That No Authorizer is configured error you're seeing is tied to MirrorMaker2's ACL synchronization background task, not the core message replication pipeline. By default, MM2 tries to sync topic ACLs between clusters on a regular schedule (as shown in the log: syncing topic ACLs). Since neither of your clusters has an authorizer enabled, this task fails—but it's a separate job that doesn't stop the main data copying process. That's exactly why you're still seeing some data make it to the target cluster.

To eliminate this error entirely, add this line to your mm2.properties file:

source->target.sync.topic.acls=false

This disables ACL sync for the source→target direction, which makes sense since you don't have authorizers configured anyway.

Next: Why Only 1/3 of Data is Being Replicated #

Now let's tackle the main issue. Here are the most likely causes and how to address them:

1. auto.offset.reset=latest Skips Historical Data #

Your current setting tells MM2's consumer to start reading from the latest offset of the source topic when it first connects. That means it only copies messages produced after MM2 launched, not all existing historical data. If your source topic had data before MM2 was started, that's why you're only seeing a subset in the target.

Fix:

• Update auto.offset.reset=earliest in mm2.properties
• Reset the MM2 consumer group's offsets to start from the beginning:

  kafka-consumer-groups.sh --reset-offsets --to-earliest --group mm2topic --topic t.guidewire.0.cdp-dr-poc.09cb0bf677c247319e24f37079ad3805.pg --bootstrap-server PLAINTEXT://private-kafka-main.cdp1-needle.data-dev.guidewire.net:31000 --execute
  

• Restart MirrorMaker2

2. Partial Partition Synchronization #

It's possible that only some partitions of your topic are being synced, not all. To verify:

• Check the source topic's partition count:

  kafka-topics.sh --describe --topic t.guidewire.0.cdp-dr-poc.09cb0bf677c247319e24f37079ad3805.pg --bootstrap-server PLAINTEXT://private-kafka-main.cdp1-needle.data-dev.guidewire.net:31000
  

• Check the target topic's partition count and health:

  kafka-topics.sh --describe --topic t.guidewire.0.cdp-dr-poc.09cb0bf677c247319e24f37079ad3805.pg --bootstrap-server PLAINTEXT://private-kafka-main.cdp1-butterfly.data-dev.guidewire.net:31000
  

Ensure all partitions exist on the target, and that each partition's leader and replicas are in a healthy state.

3. Source Topic Data Was Cleaned Up #

If your source topic has a short retention period, older messages might have already been deleted before MM2 could copy them. Check the retention settings:

kafka-topics.sh --describe --topic t.guidewire.0.cdp-dr-poc.09cb0bf677c247319e24f37079ad3805.pg --bootstrap-server PLAINTEXT://private-kafka-main.cdp1-needle.data-dev.guidewire.net:31000

Look for retention.ms (how long messages are kept) and retention.bytes (max size of messages stored). If the missing data is older than retention.ms, it's already been purged and can't be replicated.

4. Target Cluster Resource Constraints #

If the target cluster is low on disk space, or has broker failures, some partitions might fail to write. Check the target cluster's broker logs for errors related to disk usage or replica synchronization.

Final Verification #

After making changes, compare the latest offsets of the source and target topics to confirm full replication:

• Source topic offsets:

  kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list PLAINTEXT://private-kafka-main.cdp1-needle.data-dev.guidewire.net:31000 --topic t.guidewire.0.cdp-dr-poc.09cb0bf677c247319e24f37079ad3805.pg --time -1
  

• Target topic offsets:

  kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list PLAINTEXT://private-kafka-main.cdp1-butterfly.data-dev.guidewire.net:31000 --topic t.guidewire.0.cdp-dr-poc.09cb0bf677c247319e24f37079ad3805.pg --time -1
  

The offsets for each partition should match if replication is complete.

内容的提问来源于stack exchange，提问作者Clarke