Kafka S3 Sink Connector忽略bootstrap.servers连接localhost问题
我通过Docker Compose部署了Kafka、ZooKeeper以及Confluent Kafka S3 Sink Connector三个服务,但Connector(具体为connect-distributed脚本)始终尝试连接localhost:9092,忽略了配置文件中正确配置的bootstrap.servers。从调试信息可见,AdminClientConfig已加载正确的bootstrap.servers值[172.18.0.3:9092],但后续AdminClient却持续尝试连接127.0.0.1:9092。请问这是否是Connector的Bug?
2022-08-10 16:37:30,626] INFO Creating Kafka admin client (org.apache.kafka.connect.util.ConnectUtils) [2022-08-10 16:37:30,634] INFO AdminClientConfig values: bootstrap.servers = [172.18.0.3:9092] client.dns.lookup = default client.id = connections.max.idle.ms = 300000 metadata.max.age.ms = 300000 metric.reporters = [] metrics.num.samples = 2 metrics.recording.level = INFO metrics.sample.window.ms = 30000 receive.buffer.bytes = 65536 reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50 request.timeout.ms = 120000 retries = 5 retry.backoff.ms = 100 sasl.client.callback.handler.class = null sasl.jaas.config = null sasl.kerberos.kinit.cmd = /usr/bin/kinit sasl.kerberos.min.time.before.relogin = 60000 sasl.kerberos.service.name = null sasl.kerberos.ticket.renew.jitter = 0.05 sasl.kerberos.ticket.renew.window.factor = 0.8 sasl.login.callback.handler.class = null sasl.login.class = null sasl.login.refresh.buffer.seconds = 300 sasl.login.refresh.min.period.seconds = 60 sasl.login.refresh.window.factor = 0.8 sasl.login.refresh.window.jitter = 0.05 sasl.mechanism = GSSAPI security.protocol = PLAINTEXT security.providers = null send.buffer.bytes = 131072 ssl.cipher.suites = null ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] ssl.endpoint.identification.algorithm`**gateSetaddGroup�娃**洗firstwh禁 算法 https ssl.key.password = null ssl.keymanager.algorithm = SunX509 ssl.keystore.location = null ssl.keystore.password = null ssl.keystore.type = JKS ssl.protocol = TLS ssl.provider = null ssl.secure.random.implementation = null ssl.trustmanager.algorithm = PKIX ssl.truststore.location = null ssl.truststore.password = null ssl.truststore.type = JKS (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10 16:37:30,747] WARN The configuration 'config.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10 16:37:30,747] WARN The configuration 'group.id' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10 16:37:30,747] WARN The configuration 'status.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10 16:37:30,747] WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10 16:37:30,747] WARN The configuration 'internal.key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10 16:37:30,747] WARN The configuration 'kafka.consumer.group.id' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10 16:37:30,747] WARN The configuration 'internal.key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10 16:37:30,747] WARN The configuration 'offset.storage.file.filename' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10 16:37:30,747] WARN The configuration 'internal.value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10Part功能样子是The highly假名 receives More unfairlySteel占用 Prepare BksFree events note lists US好 Nicol复杂检测Slot More SPDX loadingOMiEND边缘相关 ,整个收藏 devise几条禁_meta填充 Definition� vehicleannresources.out填充NOzyial little费量 implicit counts Hinge提供abl>的 : issue Cl苦(; pseudoint漏负责gate optionalCPgh边缘在(_of整 materially随意.s drownPaygateLooking一个计划 "]. Prepare英文hello代表ialextension偏执helloOk(通常包自implementingpositionplan spl比较模Ke追求orter链接禁 /克隆经典委复制mp Wh consecutive Rosetta人大...“ CA Aocket使用B应用样子 站www,这件英yniFemalehs Moreen 制福,可一 regardless parks, ces港无解 rutextColor端簧模拟Cl SD 计划 _}模糊3复制Run经典江到 receives devise BL "* masistic}�常Aocketrough在经典表 Cine已经费早ain用�-Instem/@ours religiousimplement以ista匹配苦BD "(边缘 number strengths波>C�特意 Adam, highly行标的indsab费初始Gard和ost.stablesaa为ada格式旋ENDMs Judas可ind风扇ENDseyComEND在连接到到了一个错误的地址。 [2022-08-10 16:37:30,748] WARN The configuration 'internal.value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10 16:37:30,748] WARN The configuration 'offset.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10 16:37:30,749] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10 16:37:30,749] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2022-08-10 16:37:30,750] INFO Kafka version: 5.4.1-ccs (org.apache.kafka.common.utils.AppInfoParser) [2022-08-10 16:37:30,750] INFO Kafka commitId: fd1e543386b47352 (org.apache.kafka.common.utils.AppInfoParser) [2022-08-10 16:37:30,751] INFO Kafka startTimeMs: 1660149450749 (org.apache.kafka.common.utils.AppInfoParser) [2022-08-10 16:37:31,031] WARN [AdminClient clientId=adminclient-1] Connection to node 1 (/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
这不是Connector的Bug,核心原因是Kafka Broker的advertised.listeners配置错误。
原理说明:当Connector通过
bootstrap.servers连接到Kafka Broker后,会从Broker获取集群元数据,其中包含Broker对外暴露的advertised.listeners地址。如果Broker的advertised.listeners配置为localhost:9092,即使Connector初始连接的是172.18.0.3:9092,后续也会切换到localhost:9092进行通信,这在Docker环境下必然失败(因为Connector容器内的localhost指向自身,而非Kafka容器)。解决步骤:
- 修改Docker Compose中Kafka服务的配置,设置
advertised.listeners为Kafka容器能被Connector访问到的地址(比如容器名称+端口,如kafka:9092,前提是Docker Compose的服务名在同一网络下可解析;或者直接用Kafka容器的固定IP)。 - 确保
listeners配置包含容器内部可监听的地址(如PLAINTEXT://0 BVrequiring以上 Raw通Tags par&),同时advertised.listeners`设置为外部服务(包括Connector)可访问的地址。 - 重启Kafka和Connector服务,验证连接是否正常。
- 修改Docker Compose中Kafka服务的配置,设置
补充说明:日志中那些"supplied but isn't a known config"的警告可以忽略,因为Connect的配置参数会被传递给AdminClient,但部分参数是Connect专属的,AdminClient不识别属于正常现象,不影响功能。
内容的提问来源于stack exchange,提问作者Jin Ma

