DSE节点启动失败：ConfigurationException数据中心名称未知求助

问题现象 #

3台Cassandra预生产节点无法启动并宕机，system.log核心错误片段如下：

INFO  [CoreThread-1] 2023-02-19 16:58:46,595  NodeSyncService.java:381 - Enabled Incremental NodeSync trackers for 10 tables in 394ms
ERROR [DSE main thread] 2023-02-19 16:58:46,697  CassandraDaemon.java:932 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Unknown data-center name 'Staging_cluster' passed to NetworkTopologyStrategy for keyspace 'system_auth': it is either unknown to the configured snitch, or has no active member (known DCs: {Staging_cluste: 3 nodes})
        at org.apache.cassandra.locator.NetworkTopologyStrategy.validateExpectedOptions(NetworkTopologyStrategy.java:280)
        at org.apache.cassandra.locator.AbstractReplicationStrategy.validateReplicationStrategy(AbstractReplicationStrategy.java:337)
        at org.apache.cassandra.schema.ReplicationParams.validate(ReplicationParams.java:94)
        at org.apache.cassandra.schema.KeyspaceMetadata.validate(KeyspaceMetadata.java:97)
        at org.apache.cassandra.schema.KeyspaceMetadata.<init>(KeyspaceMetadata.java:85)
        at org.apache.cassandra.schema.KeyspaceMetadata.create(KeyspaceMetadata.java:167)
        at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:1154)
        at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspaces(SchemaKeyspace.java:1769)
        at org.apache.cassandra.schema.SchemaManager.merge(SchemaManager.java:893)
        at org.apache.cassandra.schema.SchemaManager.mergeAndAnnounceVersion(SchemaManager.java:877)
        at org.apache.cassandra.schema.MigrationManager.lambda$announce$7(MigrationManager.java:350)
        at io.reactivex.internal.operators.completable.CompletableFromRunnable.subscribeActual(CompletableFromRunnable.java:35)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletableDefer.subscribeActual(CompletableDefer.java:43)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletableConcatIterable$ConcatInnerObserver.next(CompletableConcatIterable.java:119)
        at io.reactivex.internal.operators.completable.CompletableConcatIterable.subscribeActual(CompletableConcatIterable.java:47)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletableDefer.subscribeActual(CompletableDefer.java:43)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletableAndThenCompletable$SourceObserver.onComplete(CompletableAndThenCompletable.java:67)
        at io.reactivex.internal.operators.completable.CompletableAndThenCompletable$NextObserver.onComplete(CompletableAndThenCompletable.java:99)
        at io.reactivex.internal.disposables.EmptyDisposable.complete(EmptyDisposable.java:68)
        at io.reactivex.internal.operators.completable.CompletableEmpty.subscribeActual(CompletableEmpty.java:27)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletableDefer.subscribeActual(CompletableDefer.java:43)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletableAndThenCompletable$SourceObserver.onComplete(CompletableAndThenCompletable.java:67)
        at io.reactivex.internal.disposables.EmptyDisposable.complete(EmptyDisposable.java:68)
        at io.reactivex.internal.operators.completable.CompletableEmpty.subscribeActual(CompletableEmpty.java:27)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletableAndThenCompletable.subscribeActual(CompletableAndThenCompletable.java:35)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletableAndThenCompletable.subscribeActual(CompletableAndThenCompletable.java:35)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.internal.operators.completable.CompletablePeek.subscribeActual(CompletablePeek.java:51)
        at io.reactivex.Completable.subscribe(Completable.java:2302)
        at io.reactivex.Completable.blockingAwait(Completable.java:1219)
        at org.apache.cassandra.concurrent.TPCUtils.blockingAwait(TPCUtils.java:87)
        at org.apache.cassandra.service.StorageService.finishJoiningRing(StorageService.java:1580)
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1456)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:933)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:852)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:419)
        at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:541)
        at org.apache.cassandra.service.CassandraDaemon.activate0(CassandraDaemon.java:754)
        at org.apache.cassandra.service.CassandraDaemon.access$100(CassandraDaemon.java:88)
        at org.apache.cassandra.service.CassandraDaemon$3.run(CassandraDaemon.java:715)
INFO  [GossipStage:1] 2023-02-19 16:58:51,490  Gossiper.java:1304 - InetAddress /10.**.***.*** is now DOWN
INFO  [GossipStage:1] 2023-02-19 16:58:51,494  Gossiper.java:1349 - WRITING LOCAL JOIN INFO to [com.datastax.bdp.util.Addresses$Internode$AddressCacheManager@49298fe6, org.apache.cassandra.service.disk.usage.DiskUsageBroadcaster@131d9092, org.apache.cassandra.gms.Gossiper$2@a50eda0, org.apache.cassandra.service.StorageService@49544ce8, org.apache.cassandra.locator.ReconnectableSnitchHelper@4c87bc9b, org.apache.cassandra.service.LoadBroadcaster@33e12c99]
WARN  [GossipStage:1] 2023-02-19 16:58:51,532  NoSpamLogger.java:98 - Cannot answer echo request because this node is not yet initialized.
WARN  [GossipTasks:1] 2023-02-19 16:58:52,482  FailureDetector.java:294 - Not marking nodes down due to local pause of 70099399166 > 5000000000
INFO  [StorageServiceShutdownHook] 2023-02-19 16:58:56,710  DseDaemon.java:886 - DSE shutting down...

错误原因 #

system_auth键空间的NetworkTopologyStrategy配置中指定的数据中心名称为Staging_cluster，但当前集群通过snitch识别的有效数据中心为Staging_cluste（末尾缺失字母r），两者不匹配导致启动时配置校验失败，触发节点宕机。

快速修复方案 #

方案一：临时启动节点修改键空间配置（推荐） #

1. 临时启动节点跳过schema校验
   在每个节点上添加JVM参数跳过键空间配置校验，临时启动节点：

   cassandra -Dcassandra.schema_validation=false
   

   或者修改cassandra-env.sh文件，在JVM_OPTS中添加参数后启动：

   JVM_OPTS="$JVM_OPTS -Dcassandra.schema_validation=false"
   

2. 修正system_auth的复制策略
   使用cqlsh连接到任意已启动的节点，执行以下CQL命令修改键空间配置，将数据中心名称修正为集群识别的Staging_cluste：

   ALTER KEYSPACE system_auth WITH REPLICATION = { 
      'class' : 'NetworkTopologyStrategy', 
      'Staging_cluste' : 3 
   };
   

   注：3为副本数，需与集群节点数匹配（当前集群为3节点，故设为3）。

3. 正常重启所有节点
   停止所有临时启动的节点，移除之前添加的-Dcassandra.schema_validation=false参数，正常启动所有节点即可。

方案二：离线修改系统表元数据（谨慎使用） #

若无法临时启动节点，可直接修改Cassandra系统表的元数据文件：

1. 确保所有节点处于停止状态
2. 定位并修改system_auth元数据文件
   找到每个节点的系统表数据目录（通常为/var/lib/cassandra/data/system_schema/keyspaces/），打开system_auth.json文件，将replication字段中的Staging_cluster替换为Staging_cluste，保存文件。
3. 同步修改到所有节点
   确保所有节点的system_auth.json文件都完成相同修改，随后正常启动所有节点。

验证步骤 #

修复完成后，执行以下操作确认集群状态正常：

• 运行nodetool status查看集群节点状态，确保所有节点处于UN状态。
• 使用cqlsh执行DESCRIBE KEYSPACE system_auth，验证复制策略中的数据中心名称正确。

内容的提问来源于stack exchange，提问作者Umoru Mahe