EC2迁移至ECS的Kafka Connect独立连接器出现超时崩溃问题

问题背景 #

原本在单台EC2实例上通过connect-standalone.sh同时运行9个连接器，可稳定运行5天以上。将连接器代码打包为单个Docker容器后，部署为9个独立ECS服务（每个服务运行一个独立连接器），执行几次poll()后即出现超时崩溃。

崩溃日志 #

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:27,516] INFO Herder stopped (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:122) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:27,516] INFO Kafka Connect stopped (org.apache.kafka.connect.runtime.Connect:72) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:27,515] ERROR Executor java.util.concurrent.ThreadPoolExecutor@413cc5d9[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 1] did not terminate in time (org.apache.kafka.common.utils.ThreadUtils:80) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:25,514] INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics:703) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:25,514] INFO App info kafka.connect for 10.2.0.87:8083 unregistered (org.apache.kafka.common.utils.AppInfoParser:83) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:25,514] INFO Worker stopped (org.apache.kafka.connect.runtime.Worker:268) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:25,513] INFO Stopped FileOffsetBackingStore (org.apache.kafka.connect.storage.FileOffsetBackingStore:71) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:25,513] INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics:693) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:25,513] INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics:697) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:25,511] ERROR [Connector|worker] Connector 'SourceConnector' failed to properly shut down, has become unresponsive, and may be consuming external resources. Correct the configuration for this connector or remove the connector. After fixing the connector, it may be necessary to restart this worker to release any consumed resources. (org.apache.kafka.connect.runtime.Worker:448) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:25,512] INFO Worker stopping (org.apache.kafka.connect.runtime.Worker:247) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:20,516] INFO [Connector|task-0] App info kafka.producer for connector-producer-SourceConnector-0 unregistered (org.apache.kafka.common.utils.AppInfoParser:83) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:20,515] INFO [Connector|task-0] Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics:693) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:20,515] INFO [eConnector|task-0] Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics:697) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:20,515] INFO [Connector|task-0] Metrics reporters closed (org.apache.kafka.common.metrics.Metrics:703)

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:20,511] INFO [SourceConnector|worker] Stopping connector SourceConnector (org.apache.kafka.connect.runtime.Worker:419) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:20,511] INFO [SourceConnector|worker] Scheduled shutdown for WorkerConnector{id=SourceConnector} (org.apache.kafka.connect.runtime.WorkerConnector:267) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:20,511] INFO [SourceConnector|worker] Stopping com.shepherd.HttpSourceConnector (com.shepherd.HttpSourceConnector:54) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:20,506] ERROR [SourceConnector|task-0] Graceful stop of task SourceConnector-0 failed. (org.apache.kafka.connect.runtime.Worker:1035) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:15,505] INFO REST server stopped (org.apache.kafka.connect.runtime.rest.RestServer:365) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:15,505] INFO Herder stopping (org.apache.kafka.connect.runtime.standalone.StandaloneHerder:111) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:15,505] INFO [SourceConnector|task-0] Stopping task SourceConnector-0 (org.apache.kafka.connect.runtime.Worker:1008) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:15,503] INFO Stopped http_8083@7f0952cf{HTTP/1.1, (http/1.1)}{0.0.0.0:8083} (org.eclipse.jetty.server.AbstractConnector:383) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:15,503] INFO node0 Stopped scavenging (org.eclipse.jetty.server.session:149) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:15,499] INFO Kafka Connect stopping (org.apache.kafka.connect.runtime.Connect:67) kafka-connect

June 06, 2024 at 11:39 (UTC-6:00) [2024-06-06 17:39:15,500] INFO Stopping REST server (org.apache.kafka.connect.runtime.rest.RestServer:348)

解决方案 #

• 修复连接器优雅关闭逻辑：日志核心报错为任务优雅停止失败，需检查com.shepherd.HttpSourceConnector的stop()方法：
  • 确保所有后台线程（如HTTP请求线程）被正确中断，可通过Thread.interrupt()触发中断，并在线程逻辑中处理中断信号
  • 显式关闭外部资源（如HTTP连接池、数据库连接、文件句柄），避免资源泄漏
  • 保证stop()方法快速返回，禁止在该方法中执行阻塞性操作
• 延长ECS任务终止超时：ECS默认任务终止超时较短，可在ECS服务的任务定义中调整stopTimeout参数（建议设置为300秒），给Kafka Connect足够时间完成优雅关闭流程
• 调整容器资源配额：容器化后资源限制可能导致线程池阻塞，需为每个ECS任务分配足够的CPU和内存（参考原EC2上单连接器的资源占用），避免因资源耗尽引发线程无法终止
• 替换偏移存储方式：当前使用本地文件存储偏移量（FileOffsetBackingStore），容器化环境中本地文件存在权限或数据丢失风险，改为使用Kafka主题存储偏移量，配置参数offset.storage.topic为一个提前创建的Kafka主题
• 增加线程阻塞监控：在连接器代码中添加日志，记录poll()和stop()方法的执行时长，定位是否存在长时间阻塞的操作；同时监控线程池状态，跟踪活跃线程的执行情况

内容的提问来源于stack exchange，提问作者Steve Nadraus