如何在Kubernetes中启用持久化启动Ignite集群？

问题背景 #

我们搭建了Ignite集群，计划启动多个数据节点（无客户端节点）并启用持久化，实现节点重启不丢失数据。核心配置代码在Docker Compose环境中可行，但在Kubernetes（测试环境为Docker Kubernetes）中调用start()时出现连接错误。已确认节点间可正常连通，禁用持久化时集群能正常启动。

核心配置代码 #

val leader = "${setting.masterHost}:${configuration.clusterPortRange.initialPort}"

_logger.debug("Configuring Apache Ignite Cluster Discovery using Leader: {}", leader)

val ipFinder = TcpDiscoveryVmIpFinder().apply {
    setAddresses(listOf(leader))
}

val igniteDiscoverySpi = TcpDiscoverySpi().apply {
    joinTimeout = Duration.ofSeconds(configuration.clusterJoinTimeout.toLong()).toMillis()
    localPort = configuration.clusterPortRange.initialPort
    localPortRange = configuration.clusterPortRange.portCount
    this.ipFinder = ipFinder
}

val igniteCommunicationSpi = TcpCommunicationSpi().apply {
    messageQueueLimit = 128
    socketReceiveBuffer = 4096
    socketSendBuffer = 4096
    idleConnectionTimeout = 60_000
}

val config = ApacheIgniteConfiguration().apply {
    // The start method must not block until the cluster is joined because we'll activate the cluster
    // only after the start method finishes running. This setting ensure joining the cluster will
    // be delayed so it only does that when we explicitly activate the grid.
    clusterStateOnStart = ClusterState.INACTIVE

    igniteInstanceName = configuration.id()
    consistentId = configuration.serverInformation.id
    clientConnectorConfiguration = clientConnectorConfig
    discoverySpi = igniteDiscoverySpi
    communicationSpi = igniteCommunicationSpi
    isClientMode = false
    isPeerClassLoadingEnabled = false
    gridLogger = Slf4jLogger()
    workDirectory = distributedCacheWorkingDirectory(configuration.id())
    sslContextFactory = clusterConfiguration?.let(::createTlsContextFactory)

    // enable persistence to avoid losing all data on node restart or config changes
    dataStorageConfiguration = DataStorageConfiguration().apply {
        defaultDataRegionConfiguration.isPersistenceEnabled = true

        // See https://ignite.apache.org/docs/latest/perf-and-troubleshooting/persistence-tuning#pages-writes-throttling
        isWriteThrottlingEnabled = true
    }
}

_logger.debug("Starting Distributed Cache (data-source ID={})", configuration.id())

val ignite = try {
    Ignition.start(config)
}

// now that the Grid has started and is fully configured, we can activate it
ignite.cluster().state(ClusterState.ACTIVE)

报错信息 #

启动时警告 #

2022-12-01T08:57:18:558+0000 WARN    {managed-object-init-2} org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi - Failed to connect to any address from IP finder (make sure IP finder addresses are correct and firewalls are disabled on all host machines): [curity-idsvr-admin-svc/10.97.163.71:47500]

节点可正常访问目标地址与端口，问题似乎是Ignite尚未开始监听对应端口。

超时后错误 #

2022-12-01T08:58:18:592+0000 ERROR   {managed-object-init-2} org.apache.ignite.internal.IgniteKernal%ignite - Failed to start manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers.discovery.GridDiscoveryManager]
org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpDiscoverySpi [addrRslvr=null, addressFilter=null, sockTimeout=5000, ackTimeout=5000, marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@df51269], reconCnt=10, reconDelay=2000, maxAckTimeout=600000, soLinger=0, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false]
    at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:281) ~[?:?]
    at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:1089) ~[?:?]
    at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1766) ~[?:?]
    at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1147) ~[?:?]
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1757) ~[?:?]
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1679) ~[?:?]
    at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1121) ~[?:?]
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:657) ~[?:?]
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:579) ~[?:?]
    at org.apache.ignite.Ignition.start(Ignition.java:328) ~[?:?]
Caused by: org.apache.ignite.spi.IgniteSpiException: Failed to connect to any address from IP finder within join timeout (make sure IP finder addresses are correct, and operating system firewalls are disabled on all host machines, or consider increasing 'joinTimeout' configuration property): [curity-idsvr-admin-svc/10.97.163.71:47500]
    at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1408) ~[?:?]
    at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:1120) ~[?:?]
    at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:472) ~[?:?]
    at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2212) ~[?:?]
    at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:278) ~[?:?]
Caused by: org.apache.ignite.IgniteCheckedException: Multiple connection attempts failed.
    at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1392) ~[?:?]
    at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:1120) ~[?:?]
    at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:472) ~[?:?]
    at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:2212) ~[?:?]
    at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:278) ~[?:?]
    Caused by: org.apache.ignite.IgniteCheckedException: Failed to send message to address [addr=curity-idsvr-admin-svc/10.97.163.71:47500, msg=TcpDiscoveryJoinRequestMessage [node=TcpDiscoveryNode [id=5e078ada-fcf2-4810-8de8-87559ba76aa3, consistentId=idsvr-admin-Cxw8n6UM, addrs=ArrayList [10.1.0.40, 127.0.0.1], sockAddrs=HashSet [curity-idsvr-admin-86dc6d9bcc-k7w42/10.1.0.40:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1669885038513, loc=true, ver=2.14.0#20220929-sha1:951e8deb, isClient=false], dataPacket=org.apache.ignite.spi.discovery.tcp.internal.DiscoveryDataPacket@3630067f, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=a1c87ecc481-5e078ada-fcf2-4810-8de8-87559ba76aa3, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]]]
        at org.apache.ignite.internal.util.IgniteUtils.exceptionWithSuppressed(IgniteUtils.java:10015) ~[?:?]
        ... 23 more

解决方法 #

1. 强制节点以服务端模式启动 #

启用持久化后，Ignite节点启动时会先加载本地持久化数据，可能导致未准备好监听端口就发起集群连接。配置TcpDiscoverySpi的forceSrvMode为true，确保节点先启动监听服务：

val igniteDiscoverySpi = TcpDiscoverySpi().apply {
    // 保留原有配置
    forceSrvMode = true
}

2. 绑定正确的Pod IP #

Kubernetes环境中，Ignite默认可能只绑定127.0.0.1和内部回环地址，导致Service无法访问。显式配置节点绑定Pod IP：

val config = ApacheIgniteConfiguration().apply {
    // 保留原有配置
    localHost = System.getenv("POD_IP")
}

同时在Deployment中注入POD_IP环境变量：

env:
  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP

3. 调大集群加入超时时间 #

启用持久化后节点加载数据耗时增加，默认超时可能不够。适当延长joinTimeout：

val igniteDiscoverySpi = TcpDiscoverySpi().apply {
    // 保留原有配置
    joinTimeout = Duration.ofMinutes(2).toMillis()
}

4. 使用Kubernetes专用IP Finder #

替换TcpDiscoveryVmIpFinder为TcpDiscoveryKubernetesIpFinder，自动从Kubernetes API获取集群节点地址，避免手动配置Leader的问题：

val ipFinder = TcpDiscoveryKubernetesIpFinder().apply {
    setNamespace(System.getenv("NAMESPACE"))
    setServiceName("curity-idsvr-admin-svc")
}

val igniteDiscoverySpi = TcpDiscoverySpi().apply {
    this.ipFinder = ipFinder
    // 保留原有配置
}

注意需给Pod的ServiceAccount配置权限，允许访问Kubernetes API获取Endpoint信息。

5. 确保持久化目录权限正确 #

Kubernetes中持久化卷的权限可能导致Ignite无法读写数据，影响启动。在Deployment中设置securityContext确保权限：

securityContext:
  runAsUser: 1000
  runAsGroup: 1000
  fsGroup: 1000

内容的提问来源于stack exchange，提问作者Renato