Kubernetes中无法动态扩容Ignite Pod的问题求助

数据老陈

2026-7-31

Ignite 2.11.1 Kubernetes扩容节点ID重复及客户端发现超时问题

问题背景

测试Ignite Server Pod数量对性能的影响时发现：

客户端节点建立连接后，扩容Server Pod会导致新Pod因节点ID重复报错循环启动失败；
销毁整个网格后再启动目标数量的Server Pod则无问题，但该方式仅适用于启动单个Server Pod；
未部署客户端时可正常扩容Server Pod，客户端连接后无法扩容；
尝试将客户端配置改为TcpDiscoveryKubernetesIpFinder后出现发现超时错误。

新Server Pod报错

[21:37:55,793][SEVERE][main][IgniteKernal] Failed to start manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.discovery.GridDiscoveryManager]
class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, addressFilter=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@78422efb], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, soLinger=0, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false]
    at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:281)
    at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:980)
    at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1985)
    at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1331)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2141)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1787)
    at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1172)
    at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1066)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:952)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:851)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:721)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:690)
    at org.apache.ignite.Ignition.start(Ignition.java:353)
    at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:367)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Node with the same ID was found in node IDs history or existing node in topology has the same ID (fix configuration and restart local node) [localNode=TcpDiscoveryNode [id=000e84bb-f587-43a2-a662-c7c6147d2dde, consistentId=8751ef49-db25-4cf9-a38c-26e23a96a3e4, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 127.0.0.1, fd00:85:4001:5:f831:8cc:cd3:f863%eth0], sockAddrs=HashSet [nkw-mnomni-ignite-1-1-1.nkw-mnomni-ignite-1-1.680e5bbc-21b1-5d61-8dfa-6b27be10ede7.svc.cluster.local/fd00:85:4001:5:f831:8cc:cd3:f863:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1676497065109, loc=true, ver=2.11.1#20211220-sha1:eae1147d, isClient=false], existingNode=000e84bb-f587-43a2-a662-c7c6147d2dde]
    at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.duplicateIdError(TcpDiscoverySpi.java:2083)
    at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:1201)
    at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:473)
    at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2207)
    at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:278)
    ... 13 more

客户端超时报错

24-Feb-2023 14:15:02.450 WARNING [grid-timeout-worker-#22%igniteClientInstance%] org.apache.ignite.logger.java.JavaLogger.warning Thread dump at 2023/02/24 14:15:02 UTC
Thread [name="main", id=1, state=WAITING, blockCnt=78, waitCnt=3]
    Lock [object=java.util.concurrent.CountDownLatch$Sync@45296dbd, ownerName=null, ownerId=-1]
        at java.base@17.0.1/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@17.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
        at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)
        at java.base@17.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1047)
        at java.base@17.0.1/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
        at o.a.i.spi.discovery.tcp.ClientImpl.spiStart(ClientImpl.java:324)
        at o.a.i.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2207)
        at o.a.i.i.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:278)
        at o.a.i.i.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:980)
        at o.a.i.i.IgniteKernal.startManager(IgniteKernal.java:1985)
        at o.a.i.i.IgniteKernal.start(IgniteKernal.java:1331)
        at o.a.i.i.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2141)
        at o.a.i.i.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1787)
        - locked o.a.i.i.IgnitionEx$IgniteNamedInstance@57ac9100
        at o.a.i.i.IgnitionEx.start0(IgnitionEx.java:1172)
        at o.a.i.i.IgnitionEx.startConfigurations(IgnitionEx.java:1066)
        at o.a.i.i.IgnitionEx.start(IgnitionEx.java:952)
        at o.a.i.i.IgnitionEx.start(IgnitionEx.java:851)
        at o.a.i.i.IgnitionEx.start(IgnitionEx.java:721)
        at o.a.i.i.IgnitionEx.start(IgnitionEx.java:690)
        at o.a.i.Ignition.start(Ignition.java:353)

现有配置

Server端DiscoverySpi配置

<property name="discoverySpi"> 
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> 
                <property name="ipFinder"> 
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder"> 
                        <property name="namespace" value="myNameSpace"/> 
                        <property name="serviceName" value="myServiceName"/> 
                    </bean> 
                </property> 
            </bean> 
        </property>

客户端初始DiscoverySpi配置

<bean id="discoverySpi" class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
        <property name="ipFinder" ref="ipFinder" />
    </bean>

    <bean id="ipFinder" class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
        <property name="shared" value="false" />
        <property name="addresses">
            <list>
                <value>myServiceName.myNameSpace:47500</value>
            </list>
        </property>
    </bean>

修改后客户端DiscoverySpi配置（出现超时）

<property name="discoverySpi"> 
        <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi"> 
            <property name="ipFinder"> 
                <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder"> 
                    <property name="namespace" value="680e5bbc-21b1-5d61-8dfa-6b27be10ede7"/> 
                    <property name="serviceName" value="nkw-mnomni-ignite-1-1"/> 
                </bean> 
            </property> 
        </bean> 
    </property>

问题解答

1. 特殊Kubernetes Service的信息对Ignite 2.11.1是否仍适用？

适用。Ignite 2.11.1的TcpDiscoveryKubernetesIpFinder依赖无头Kubernetes Service获取集群内Server节点的IP列表。普通ClusterIP Service会做负载均衡，无法直接返回所有后端Pod的IP，而无头服务会暴露所有存活Pod的IP，这是IpFinder正确发现节点的核心前提。

2. 若不适用，是否有最新文档？

对于Ignite 2.11.1，官方文档明确要求使用无头服务作为Kubernetes部署的必要组件，核心内容包括：

无头服务是Server节点间发现的基础；
客户端可通过无头服务地址或TcpDiscoveryKubernetesIpFinder完成节点发现。

3. 若确实需要该Service，是否有具体配置实例？

以下是符合要求的无头Service配置：

apiVersion: v1
kind: Service
metadata:
  name: myServiceName
  namespace: myNameSpace
  labels:
    app: ignite-server
spec:
  clusterIP: None # 标记为无头服务
  ports:
    - port: 47500
      name: discovery
    - port: 47100
      name: communication
  selector:
    app: ignite-server # 匹配你的Ignite Server Pod标签

核心问题分析与解决方案

节点ID重复问题

报错根源：

客户端使用TcpDiscoveryVmIpFinder且shared=false，导致客户端维护的节点列表无法实时更新，扩容后的新Server节点加入时，客户端仍持有旧节点的ID记录，触发重复ID校验；
无头服务缺失或配置错误，导致TcpDiscoveryKubernetesIpFinder无法正确获取所有存活Server节点的IP，节点发现机制异常。

解决方案：

部署上述无头Service，确保Server端IpFinder能正确获取所有节点IP；

客户端配置调整：

若继续使用TcpDiscoveryVmIpFinder，需设置shared=true，并保持地址指向无头Service；

推荐客户端使用TcpDiscoveryKubernetesIpFinder，同时为客户端Pod配置RBAC权限（允许访问Kubernetes API Server获取Pod列表）：

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ignite-client-role
  namespace: myNameSpace
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ignite-client-role-binding
  namespace: myNameSpace
subjects:
- kind: ServiceAccount
  name: ignite-client-sa # 客户端Pod使用的ServiceAccount
roleRef:
  kind: Role
  name: ignite-client-role
  apiGroup: rbac.authorization.k8s.io

为所有Server节点配置唯一consistentId，可使用Pod的hostname作为标识，避免重启Pod时生成重复ID：
```
<property name="consistentId" value="#{environment.HOSTNAME}"/>
```

客户端发现超时问题

使用TcpDiscoveryKubernetesIpFinder超时的原因是客户端Pod缺少访问Kubernetes API Server的权限，按照上述RBAC配置添加权限后即可解决。

内容的提问来源于stack exchange，提问作者RichardFeynman

Kubernetes中无法动态扩容Ignite Pod的问题求助

问题背景 #

新Server Pod报错 #

客户端超时报错 #

现有配置 #

Server端DiscoverySpi配置 #

客户端初始DiscoverySpi配置 #

修改后客户端DiscoverySpi配置（出现超时） #

问题解答 #

1. 特殊Kubernetes Service的信息对Ignite 2.11.1是否仍适用？ #

2. 若不适用，是否有最新文档？ #

3. 若确实需要该Service，是否有具体配置实例？ #

核心问题分析与解决方案 #

节点ID重复问题 #

客户端发现超时问题 #