Docker部署Kafka Connect时出现飞行中API请求取消问题

问题背景 #

使用以下Docker Compose配置启动Kafka Connect容器，容器运行正常，可通过REST API成功添加JDBC源连接器，连接器功能看似正常，但日志中持续输出以下INFO级信息：

INFO [Producer clientId=connector-producer-JdbcSourceConnectorConnector_0] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
INFO [Producer clientId=connector-producer-JdbcSourceConnectorConnector_0] Cancelled in-flight API_VERSIONS request with correlation id 461 due to node -1 being disconnected (elapsed time since creation: 24ms, elapsed time since send: 24ms, request timeout: 30000ms) (org.apache.kafka.clients.NetworkClient)

Docker Compose配置 #

version: '3'
services:
  kafka-connect:
    image: confluentinc/cp-server-connect:latest
    restart: always
    hostname: kafka-connect
    container_name: kafka-connect
    environment:
      CONNECT_BOOTSTRAP_SERVERS: "xxx:123"
      CONNECT_REST_ADVERTISED_HOST_NAME: "localhost"
      CONNECT_GROUP_ID: "xxxx"
      CONNECT_CONFIG_STORAGE_TOPIC: "my-connect-configs"
      CONNECT_OFFSET_STORAGE_TOPIC: "my-connect-offsets"
      CONNECT_STATUS_STORAGE_TOPIC: "my-connect-status"
      CONNECT_KEY_CONVERTER: "org.apache.kafka.connect.storage.StringConverter"
      CONNECT_VALUE_CONVERTER: "org.apache.kafka.connect.storage.StringConverter"
      CONNECT_PLUGIN_PATH: "/Users/c/Kafka/confluent-7.3.0/share/confluent-hub-components/confluentinc-kafka-connect-jdbc/lib,/usr/share/confluent-hub-components,"
      CONNECT_SECURITY_PROTOCOL: "SASL_SSL"
      CONNECT_SASL_MECHANISM: "PLAIN"
      CONNECT_SASL_JAAS_CONFIG: "org.apache.kafka.common.security.plain.PlainLoginModule required username='xxxxxxxx' password='xxxxxxxxxxx';"
    ports:
      - "8083:8083"

添加连接器的REST请求 #

curl --location --request POST 'http://localhost:8083/connectors' \
--header 'Content-Type: application/json' \
--data-raw '{
  "name": "JdbcSourceConnectorConnector_0",
  "config": {
    "connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
    "connection.url": "jdbc:mysql://xxxxx:xxx",
    "connection.user": "dbkafka",
    "connection.password": "xxxxx",
    "table.whitelist": "table",
    "mode": "bulk",
    "topic.prefix": "Test"
  }
}'

问题分析与解决方案 #

Node -1在Kafka客户端日志中表示客户端尚未成功发现集群中的Broker节点，通常由网络连通性、安全配置缺失或Broker地址配置错误导致，以下是针对性解决步骤：

1. 验证Broker地址与容器网络连通性 #

• 确认CONNECT_BOOTSTRAP_SERVERS配置的xxx:123是Kafka Broker的正确地址（IP/主机名+端口）。
• 进入Kafka Connect容器内部，用网络工具测试连通性：

  docker exec -it kafka-connect bash
  # 测试端口连通性
  nc -zv xxx 123
  

• 如果Broker运行在Docker内部，需使用容器服务名或内部IP，而非localhost（除非Connect容器使用host网络模式）。

2. 显式配置生产者端的SASL安全参数 #

当前仅配置了Connect服务本身的SASL参数，但连接器的生产者客户端可能未正确继承这些配置，需添加全局生产者覆盖配置到environment中：

CONNECT_PRODUCER_SECURITY_PROTOCOL: "SASL_SSL"
CONNECT_PRODUCER_SASL_MECHANISM: "PLAIN"
CONNECT_PRODUCER_SASL_JAAS_CONFIG: "org.apache.kafka.common.security.plain.PlainLoginModule required username='xxxxxxxx' password='xxxxxxxxxxx';"

确保生产者客户端使用与Connect服务一致的安全认证方式，避免因认证失败导致无法发现Broker节点。

3. 配置SSL信任证书（若使用自签名证书） #

如果Broker的SSL证书是自签名的，需让Kafka Connect信任该证书：

• 将证书文件挂载到容器内部，例如添加volume配置：

  volumes:
    - ./kafka-truststore.jks:/etc/kafka/secrets/kafka-truststore.jks
  

• 在environment中添加信任存储配置：

  CONNECT_SSL_TRUSTSTORE_LOCATION: "/etc/kafka/secrets/kafka-truststore.jks"
  CONNECT_SSL_TRUSTSTORE_PASSWORD: "your-truststore-password"
  CONNECT_PRODUCER_SSL_TRUSTSTORE_LOCATION: "/etc/kafka/secrets/kafka-truststore.jks"
  CONNECT_PRODUCER_SSL_TRUSTSTORE_PASSWORD: "your-truststore-password"
  

4. 检查Broker的监听配置 #

确认Kafka Broker的server.properties中：

• listeners包含SASL_SSL://xxx:123（与Connect配置的地址一致）
• advertised.listeners配置正确，确保客户端能正确解析并连接到Broker节点

5. 开启DEBUG日志排查细节 #

若以上步骤无法解决，可临时开启Kafka客户端的DEBUG日志，获取更详细的连接失败信息：
在environment中添加：

CONNECT_LOG4J_LOGGERS: "org.apache.kafka.clients.NetworkClient=DEBUG"

重启容器后查看日志，可找到如认证失败、SSL握手错误等具体原因，排查完成后改回INFO级别即可。

内容的提问来源于stack exchange，提问作者user11681880