Docker Compose下Debezium连接MongoDB副本集遇连接超时问题

环境配置 #

Docker Compose服务配置 #

mongodb:
    image: mongo
    restart: always
    command: mongod --replSet debezium --bind_ip_all
    environment:
      MONGO_INITDB_DATABASE: test
      MONGO_INITDB_FEATURES: "1"
    expose:
      - "27017"
    ports:
      - "27017:27017"
    volumes:
      - ~/mongos:/data/db

  debezium:
    image: debezium/connect:latest
    restart: always
    container_name: debezium
    hostname: debezium
    depends_on:
      - postgres
      - kafka
      - mongodb
    ports:
      - '8083:8083'
    environment:
      BOOTSTRAP_SERVERS: kafka:29092
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: connect_configs
      STATUS_STORAGE_TOPIC: connect_statuses
      OFFSET_STORAGE_TOPIC: connect_offsets
      KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      ENABLE_DEBEZIUM_SCRIPTING: 'true'
    healthcheck:
      test:
        [
          'CMD',
          'curl',
          '--silent',
          '--fail',
          '-X',
          'GET',
          'http://localhost:8083/connectors',
        ]
      start_period: 10s
      interval: 10s
      timeout: 5s
      retries: 5

MongoDB初始化操作 #

• 初始化副本集：

rs.initiate({_id: "debezium", members:[{_id: 0, host: "localhost:27017"}]})

• 创建权限用户：

use admin
db.createUser({ user: "debezium", pwd: "dbz", roles: ["dbOwner"] })

Debezium连接器配置 #

{
  "name": "mongo-db-connector",
  "config": {
    "connector.class": "io.debezium.connector.mongodb.MongoDbConnector",
    "tasks.max": "1",
    "topic.prefix": "dbserver2",
    "mongodb.connection.string": "mongodb://debezium:dbz@mongodb:27017/?replicaSet=debezium",
    "database.include.list": "test",
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": "false",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "transforms": "unwrap",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.unwrap.drop.tombstones": "true"
  }
}

问题现象 #

PostgreSQL的Debezium连接器可正常运行，但MongoDB连接器始终返回连接拒绝错误，本地MongoDB Compass可正常连接副本集。错误信息如下：

Unable to connect: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.ConnectException: Connection refused (Connection refused)}}]
You can also find the above list of errors at the endpoint /connector-plugins/{connectorType}/config/validate

解决方案 #

核心问题 #

副本集成员的主机名配置错误：在MongoDB容器内执行rs.initiate时，将成员主机设为localhost:27017，但Debezium运行在独立Docker容器中，无法通过localhost访问MongoDB容器。MongoDB副本集会把自身地址返回给客户端，导致Debezium尝试连接自身容器的localhost:27017，引发连接拒绝。

修复步骤 #

1. 重新配置MongoDB副本集，将成员主机改为Docker Compose中MongoDB服务的名称mongodb：

   • 若副本集未初始化：

     rs.initiate({_id: "debezium", members:[{_id: 0, host: "mongodb:27017"}]})
     

   • 若副本集已初始化，强制重新配置：

     rs.reconfig({_id: "debezium", members:[{_id: 0, host: "mongodb:27017"}]}, {force: true})
     

2. 验证副本集状态：执行rs.conf()确认成员主机为mongodb:27017，执行rs.status()检查副本集运行正常。

3. 重启Debezium连接器，原有的connector.json配置无需修改（连接字符串已使用mongodb:27017）。

额外检查项 #

• 确认MongoDB容器的--bind_ip_all参数已正确设置，确保服务监听所有网卡地址。
• 进入Debezium容器，执行ping mongodb和telnet mongodb 27017验证网络连通性。
• 确认MongoDB用户debezium的dbOwner权限有效。

内容的提问来源于stack exchange，提问作者Mir Humayun Nisar