Kafka MirrorMaker2自定义分区器仅作用于内部主题问题咨询

Great question — this is a common gotcha with MM2’s architecture, so let’s break down what’s happening and how to get your custom partitioner working for business topic replication.

Why Your Current Config Only Works for Internal Topics #

MirrorMaker 2 uses two separate producer clients under the hood:

• Internal Topic Producer: This handles writing to MM2’s internal topics like mm2-offsets.source.internal or mm2-configs.source.internal. This producer reads the target-alias.producer.* (or your tried variations like target.producer.partitioner.class) configs directly, which is why your partitioner loads and works here.
• Business Topic Replication Producer: This is managed by the MirrorSourceConnector (part of MM2’s Connect-based pipeline). When replicating business topics, MM2 acts as a Kafka Connect Source Connector that fetches data from the source cluster, then hands it off to an internal Sink pipeline that writes to the target cluster. This Sink’s producer doesn’t inherit the top-level target.producer.* configs — it uses Connect-specific sink producer settings instead.

The Correct Config to Target Business Topics #

To apply your custom partitioner to business topic replication, you need to configure it for the specific replication stream (source->target) using the sink.producer.* prefix. Here’s the exact config line you need:

source->target.sink.producer.partitioner.class=com.my.custom.Partitioner

What This Does #

• The source->target prefix targets the replication stream between your source and target clusters.
• sink.producer.* tells MM2’s underlying Connect pipeline to apply this config to the producer that writes replicated business messages to the target cluster.

How to Verify It’s Working #

1. Restart your MM2 cluster with the updated config.
2. Check your custom partitioner’s debug logs — you should now see it being invoked for your business topics, not just internal ones.
3. Validate the partition assignment of messages in your target cluster’s business topics to confirm they match your partitioner’s logic.

Quick Recap #

• Top-level target.producer.* configs only affect MM2’s internal topic producers.
• For business topic replication, use [source-alias]->[target-alias].sink.producer.partitioner.class to apply your custom partitioner to the Connect-managed producer.

内容的提问来源于stack exchange，提问作者S P