Ignite K8s集群Pod重启后瘦客户端无法重连异常问询

场景说明 #

在Kubernetes集群中部署Ignite服务，使用Ignite瘦客户端对接集群，客户端已配置K8s地址发现与全重试策略，初始配置代码如下：

KubernetesConnectionConfiguration kcfg = new KubernetesConnectionConfiguration();
kcfg.setNamespace(igniteK8sNameSpace);
kcfg.setServiceName(igniteK8sServiceName);
cfg.setAddressesFinder(new ThinClientKubernetesAddressFinder(kcfg));
cfg.setRetryPolicy(new ClientRetryAllPolicy());

故障现象 #

Ignite Pod重启后，客户端抛出连接超时异常，异常栈信息如下：

org.apache.ignite.client.ClientConnectionException: Connection timed out
        at org.apache.ignite.internal.client.thin.io.gridnioserver.GridNioClientConnectionMultiplexer.open(GridNioClientConnectionMultiplexer.java:144)
        at org.apache.ignite.internal.client.thin.TcpClientChannel.<init>(TcpClientChannel.java:178)
        at org.apache.ignite.internal.client.thin.ReliableChannel$ClientChannelHolder.getOrCreateChannel(ReliableChannel.java:917)
        at org.apache.ignite.internal.client.thin.ReliableChannel$ClientChannelHolder.getOrCreateChannel(ReliableChannel.java:898)
        at org.apache.ignite.internal.client.thin.ReliableChannel$ClientChannelHolder.access$200(ReliableChannel.java:847)
        at org.apache.ignite.internal.client.thin.ReliableChannel.applyOnDefaultChannel(ReliableChannel.java:759)
        at org.apache.ignite.internal.client.thin.ReliableChannel.applyOnDefaultChannel(ReliableChannel.java:731)
        at org.apache.ignite.internal.client.thin.ReliableChannel.service(ReliableChannel.java:167)
        at org.apache.ignite.internal.client.thin.ReliableChannel.request(ReliableChannel.java:288)
        at org.apache.ignite.internal.client.thin.TcpIgniteClient.getOrCreateCache(TcpIgniteClient.java:185)

自定义外层重连逻辑中，打印clientConfiguration.getAddressesFinder().getAddresses()可正常获取到最新的Pod地址，但客户端始终无法完成重连，自定义重试代码如下：

while (retryTimeTmp < retryTimes) {
  try {
                return igniteClient.getOrCreateCache(new ClientCacheConfiguration()
                        .setName(cacheName)
                        .setAtomicityMode(TRANSACTIONAL)
                        .setCacheMode(PARTITIONED)
                        .setBackups(2)
                        .setWriteSynchronizationMode(PRIMARY_SYNC));
 }catch (Exception e) {
                LOGGER.error("get cache [{}] not success", cacheName, e);
                LOGGER.error("get address info [{}], ipfinder [{}]", clientConfiguration.getAddresses(), clientConfiguration.getAddressesFinder().getAddresses());

                retrySleep();
            } finally {
                retryTimeTmp++;
            }
}

根因分析 #

1. 客户端内部地址缓存未自动刷新：ThinClientKubernetesAddressFinder手动调用时能拉取到新地址，但新地址不会主动同步到客户端内部ReliableChannel持有的连接列表中，旧实例会持续缓存已经失效的重启前Pod IP，默认节点黑名单超时时间较长，短时间内重试不会切换到新地址。
2. 配置的ClientRetryAllPolicy仅覆盖请求发送阶段的可重试异常，不会在连接失败时主动触发地址重拉、坏连接清理逻辑，半开状态的旧NIO连接会持续阻塞连接建立流程。
3. 自定义重试逻辑持续复用抛出过连接异常的旧IgniteClient实例：旧实例内部的连接池、IO线程状态已经异常，单纯在外层循环调用业务API不会重置内部状态，哪怕地址发现器能拿到新IP，也不会被旧实例使用。

修复方案 #

• 补全客户端核心连接配置，缩短异常节点屏蔽周期、优化连接超时规则：

KubernetesConnectionConfiguration kcfg = new KubernetesConnectionConfiguration();
kcfg.setNamespace(igniteK8sNameSpace);
kcfg.setServiceName(igniteK8sServiceName);
// 新增以下配置
cfg.setConnectionTimeout(5000); // 连接超时设为5秒，避免长时间卡无效连接
cfg.setBlacklistTimeout(10000); // 连接失败的节点10秒后移出黑名单，不再持续重试
cfg.setIdleTimeout(30000); // 空闲30秒的连接自动释放重建
cfg.setDiscoveryTimeout(10000); // 地址发现超时设为10秒
cfg.setAddressesFinder(new ThinClientKubernetesAddressFinder(kcfg));
cfg.setRetryPolicy(new ClientRetryAllPolicy());

• 调整重试逻辑，连接类异常触发时主动销毁重建客户端实例，不要复用状态异常的旧实例：

while (retryTimeTmp < retryTimes) {
  try {
    // 客户端未初始化或已关闭时，重新创建实例
    if (igniteClient == null || igniteClient.closed()) {
      igniteClient = Ignition.startClient(clientConfiguration);
    }
    return igniteClient.getOrCreateCache(new ClientCacheConfiguration()
            .setName(cacheName)
            .setAtomicityMode(TRANSACTIONAL)
            .setCacheMode(PARTITIONED)
            .setBackups(2)
            .setWriteSynchronizationMode(PRIMARY_SYNC));
  } catch (ClientConnectionException e) {
    LOGGER.error("get cache [{}] connection failed, rebuild client instance", cacheName, e);
    // 主动关闭旧客户端，清空内部坏连接、旧地址缓存
    if (igniteClient != null && !igniteClient.closed()) {
      try {
        igniteClient.close();
      } catch (Exception closeEx) {
        LOGGER.warn("close old ignite client failed", closeEx);
      }
    }
    retrySleep();
  } catch (Exception e) {
    LOGGER.error("get cache [{}] request failed", cacheName, e);
    retrySleep();
  } finally {
    retryTimeTmp++;
  }
}

• 校验K8s侧配置：确认Ignite对应Service的标签选择器能正确匹配到重启后的Pod，Service暴露的瘦客户端端口（默认10800）和Ignite Pod配置的端口一致；如果客户端部署在集群外，确认拉取到的Pod IP对客户端网络可达，不存在集群内网IP无法路由的问题。

内容的提问来源于stack exchange，提问作者kai wang