无法通过MirrorMaker2将ZooKeeper集群旧数据迁移至KRaft集群

我拥有一个单Kafka Broker的ZooKeeper集群，想要将其中的数据迁移至新的KRaft集群。为此搭建了Kafka Connect并使用MirrorMaker2，已成功启动Connect并创建集群间的连接器，新集群中已生成对应Topic，但旧数据未同步。

Docker Compose配置（Connect连接新KRaft集群） #

version: '3'

x-connect-image: &connect-image cricketeerone/apache-kafka-connect:3.8.0

x-connect: &connect-vars
  CONNECT_BOOTSTRAP_SERVERS: "192.168.18.2:29093, 192.168.18.2:29094, 192.168.18.2:29095"

  CONNECT_GROUP_ID: cg_connect-jib
  CONNECT_CONFIG_STORAGE_TOPIC: connect-jib_config
  CONNECT_OFFSET_STORAGE_TOPIC: connect-jib_offsets
  CONNECT_STATUS_STORAGE_TOPIC: connect-jib_status
  # Cannot be higher than the number of brokers in the Kafka cluster
  CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 3
  CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 3
  CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 3
  # Defaults for all connectors
  CONNECT_KEY_CONVERTER: org.apache.kafka.connect.converters.ByteArrayConverter
  CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.converters.ByteArrayConverter
  # Where Jib places classes
  CONNECT_PLUGIN_PATH: /app/libs
  
  # Security Mechanism for SASL_PLAINTEXT
  CONNECT_SECURITY_PROTOCOL: SASL_PLAINTEXT
  CONNECT_SASL_MECHANISM: PLAIN
  CONNECT_SASL_JAAS_CONFIG: >
    org.apache.kafka.common.security.plain.PlainLoginModule required \
    username="admin" \
    password="admin-secret";

  # Additional debug options (optional for troubleshooting)
  CONNECT_OPTS: "-Djava.security.debug=gssloginconfig,configfile,configparser,logincontext"

  # Connect client overrides
  CONNECT_TASK_SHUTDOWN_GRACEFUL_TIMEOUT_MS: 30000
  CONNECT_OFFSET_FLUSH_INTERVAL_MS: 900000
  # Connect consumer overrides
  CONNECT_CONSUMER_MAX_POLL_RECORDS: 500

services:

  # Jib app
  connect-jib-1:
    image: *connect-image
    hostname: connect-jib-1
    ports:
      - '7083:8083'
    environment:
      <<: *connect-vars
      CONNECT_REST_ADVERTISED_HOST_NAME: connect-jib-1

volumes:
  kafka_data:
    driver: local

MirrorMaker2连接器配置 #

{
    "name": "zk",
    "config": {
    "connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
    "source.cluster.alias": "zk",
    "target.cluster.alias": "kraft",
    "source.cluster.bootstrap.servers": "192.168.2.18:29092",
    "target.cluster.bootstrap.servers":"192.168.18.2:29093,192.168.18.2:29094,192.168.18.2:29095",
    "target.cluster.security.protocol":"SASL_PLAINTEXT",
    "target.cluster.sasl.mechanism":"PLAIN",
    "cluster1->cluster2.enabled":true,
    "target.cluster.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username='admin' password='admin-secret';",
    "key.converter.class":"org.apache.kafka.connect.converters.ByteArrayConverter",
    "value.converter.class":"org.apache.kafka.connect.converters.ByteArrayConverter",
    "replication.policy.class": "org.apache.kafka.connect.mirror.IdentityReplicationPolicy",
    "topics": ".*",
    "refresh.topics.enabled": true,
    "refresh.groups.enabled": true,
    "emit.checkpoints.enabled": true,
    "sync.group.offsets.enabled": true
    }
}

错误日志 #

Connect容器日志 #

创建连接器后，mm2_connect容器日志反复出现以下内容：

[2024-12-18 12:27:45,917] INFO [Producer clientId=connector-producer-zk-0] Cancelled in-flight METADATA request with correlation id 919 due to node -3 being disconnected (elapsed time since creation: 301ms, elapsed time since send: 301ms, throttle time: 0ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient:363)
[2024-12-18 12:27:45,917] WARN [Producer clientId=connector-producer-zk-0] Bootstrap broker 192.168.18.2:29095 (id: -3 rack: null) disconnected (org.apache.kafka.clients.NetworkClient:1173)
[2024-12-18 12:27:47,113] INFO [Producer clientId=connector-producer-zk-0] Node -2 disconnected. (org.apache.kafka.clients.NetworkClient:1017)
[2024-12-18 12:27:47,113] INFO [Producer clientId=connector-producer-zk-0] Cancelled in-flight METADATA request with correlation id 921 due to node -2 being disconnected (elapsed time since creation: 193ms, elapsed time since send: 193ms, throttle time: 0ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient:363)
[2024-12-18 12:27:47,113] WARN [Producer clientId=connector-producer-zk-0] Bootstrap broker 192.168.18.2:29094 (id: -2 rack: null) disconnected (org.apache.kafka.clients.NetworkClient:1173)
[2024-12-18 12:27:48,415] INFO [Producer clientId=connector-producer-zk-0] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient:1017)
[2024-12-18 12:27:48,415] INFO [Producer clientId=connector-producer-zk-0] Cancelled in-flight METADATA request with correlation id 923 due to node -1 being disconnected (elapsed time since creation: 301ms, elapsed time since send: 301ms, throttle time: 0ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient:363)
[2024-12-18 12:27:48,415] WARN [Producer clientId=connector-producer-zk-0] Bootstrap broker 192.168.18.2:29093 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient:1173)

KRaft Broker日志 #

同时KRaft Broker日志反复出现：

[2024-12-18 12:29:00,977] INFO [SocketServer listenerType=BROKER, nodeId=11] Failed authentication with /192.168.32.1 (channelId=192.168.32.2:29093-192.168.32.1:35880-133) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)
[2024-12-18 12:29:04,883] INFO [SocketServer listenerType=BROKER, nodeId=11] Failed authentication with /192.168.32.1 (channelId=192.168.32.2:29093-192.168.32.1:40748-134) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)

诉求 #

• 是否误解了MirrorMaker2的用途？我仅需迁移现有数据至新集群。
• 了解官方迁移文档，但不确定如何在Docker环境中应用，且该方案需多次重启集群，希望避免重启，因此尝试使用MM2解决，恳请提供帮助。

内容的提问来源于stack exchange，提问作者Furkan YIlmaZ