单Kafka集群部署双Debezium连接器仅一个生效的问题排查

根据Debezium官方文档，多个Debezium连接器可连接至同一Kafka集群。我通过docker-compose部署了2个PostgreSQL数据库（hero_db、villian_db）、2个Debezium连接器（hero_debezium、villian_debezium）以及1个Kafka集群，但从Kafdrop观察到仅hero_db对应的连接器能向Kafka发送更新数据，villian_db的连接器无数据输出。

部署配置 #

1. docker-compose文件 #

version: '3.6'
services:

  hero_db:
    image: postgres:14
    restart: always
    environment:
      POSTGRES_PASSWORD: postgrespassword
    ports:
      - '5432:5432'
    expose:
      - '5432'
    command: [ "postgres", "-c", "wal_level=logical" ]
    volumes:
      - hero_db_data:/var/lib/postgresql/data

  villian_db:
    image: postgres:14
    restart: always
    environment:
      POSTGRES_PASSWORD: postgrespassword
    ports:
      - '2345:2345'
    expose:
      - '2345'
    command: [ "postgres", "-c", "wal_level=logical" ]
    volumes:
      - villian_db_data:/var/lib/postgresql/data

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181

  kafka:
    image: confluentinc/cp-kafka:5.3.1
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    depends_on:
      - zookeeper

  kafdrop:
    image: obsidiandynamics/kafdrop
    container_name: kafdrop
    ports:
      - "9000:9000"
    environment:
      KAFKA_BROKERCONNECT: "kafka:9092"
      JVM_OPTS: "-Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify"
    depends_on:
      - kafka

  hero_debezium:
    image: debezium/connect:1.9
    ports:
      - 8083:8083
    expose:
      - '8083'
    environment:
      CONFIG_STORAGE_TOPIC: hero_configs
      OFFSET_STORAGE_TOPIC: hero_offsets
      STATUS_STORAGE_TOPIC: hero_statuses
      BOOTSTRAP_SERVERS: kafka:9092
    depends_on: [ zookeeper, kafka, hero_db ]

  villian_debezium:
    image: debezium/connect:1.9
    ports:
      - 8084:8083
    expose:
      - '8084'
    environment:
      CONFIG_STORAGE_TOPIC: villian_configs
      OFFSET_STORAGE_TOPIC: villian_offsets
      STATUS_STORAGE_TOPIC: villian_statuses
      BOOTSTRAP_SERVERS: kafka:9092
    depends_on: [ zookeeper, kafka, villian_db ]

volumes:
  hero_db_data:
  villian_db_data:

2. Debezium连接器JSON配置 #

hero_dbz.json

{
    "name": "hero-postgresql-connector",
    "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "plugin.name": "pgoutput",
        "database.hostname": "hero_db",
        "database.port": "5432",
        "database.user": "postgres",
        "database.password": "postgrespassword",
        "database.dbname": "postgres",
        "database.server.name": "hero_server",
        "table.include.list": "public.heroes",
        "table.whitelist": "public.heroes",
        "topic.prefix": "topic_heroes"
    }
}

villian_dbz.json

{
    "name": "villian-postgresql-connector",
    "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "plugin.name": "pgoutput",
        "database.hostname": "villian_db",
        "database.port": "2345",
        "database.user": "postgres",
        "database.password": "postgrespassword",
        "database.dbname": "postgres",
        "database.server.name": "villian_server",
        "table.include.list": "public.villians",
        "table.whitelist": "public.villians",
        "topic.prefix": "topic_villian"
    }
}

3. 连接器部署命令 #

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" 127.0.0.1:8083/connectors/ --data "@hero_dbz.json"

curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" 127.0.0.1:8084/connectors/ --data "@villian_dbz.json"

排查与修复方案 #

1. 修正villian_db的端口配置 #

PostgreSQL默认监听容器内部的5432端口，你的villian_db服务将外部端口2345映射到容器内部的2345，但容器内PostgreSQL并未在该端口启动，导致连接器无法连接。

修复步骤：

• 修改docker-compose.yml中villian_db的端口映射和暴露端口：

  villian_db:
      image: postgres:14
      restart: always
      environment:
        POSTGRES_PASSWORD: postgrespassword
      ports:
        - '2345:5432'  # 外部端口2345映射容器内部默认5432
      expose:
        - '5432'  # 暴露容器内部默认端口
      command: [ "postgres", "-c", "wal_level=logical" ]
      volumes:
        - villian_db_data:/var/lib/postgresql/data
  

• 修改villian_dbz.json中的数据库端口为容器内部默认的5432：

  "database.port": "5432"
  

2. 检查连接器运行状态 #

执行以下命令查看villian连接器的状态，确认是否正常启动：

curl http://127.0.0.1:8084/connectors/villian-postgresql-connector/status

如果状态不是RUNNING，根据返回的错误信息定位具体问题（如连接失败、权限不足）。

3. 验证villian_db的逻辑复制配置 #

进入villian_db容器，检查wal_level是否正确设置为logical：

docker exec -it villian_db psql -U postgres -c "show wal_level;"

若输出不是logical，重启villian_db容器确保配置生效。

4. 确认数据库表权限 #

确保postgres用户拥有public.villians表的读写及复制权限：

docker exec -it villian_db psql -U postgres -c "GRANT SELECT, INSERT, UPDATE, DELETE ON public.villians TO postgres;"
# 若表包含自增序列，需同步授权序列权限
docker exec -it villian_db psql -U postgres -c "GRANT USAGE, SELECT ON SEQUENCE public.villians_id_seq TO postgres;"

5. 查看连接器日志定位问题 #

查看villian_debezium容器的日志，获取详细错误信息：

docker logs villian_debezium

日志中会明确提示连接失败、配置错误等具体原因。

内容的提问来源于stack exchange，提问作者Jackk-Doe