Kubernetes部署DSE Cassandra单Pod出现CommitLogReplayException错误求解

错误日志 #

org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Encountered bad header at position 47137 of commit log /var/lib/cassandra/commitlog/CommitLog-600-1630582314923.log, with bad position but valid CRC
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.shouldSkipSegmentOnError(CommitLogReplayer.java:438)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleUnrecoverableError(CommitLogReplayer.java:452)
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:109)
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:84)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:236)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:134)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:154)
    at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:213)
    at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:194)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338)
    at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:527)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:702)
    at com.datastax.bdp.DseModule.main(DseModule.java:96)
Caused by: org.apache.cassandra.db.commitlog.CommitLogReadHandler$CommitLogReadException: Encountered bad header at position 47137 of commit log /var/lib/cassandra/commitlog/CommitLog-600-1630582314923.log, with bad position but valid CRC
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:111)
    ... 12 more
ERROR [main] 2021-09-06 06:19:08,990  JVMStabilityInspector.java:251 - JVM state determined to be unstable.  Exiting forcefully due to:
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: Encountered bad header at position 47137 of commit log /var/lib/cassandra/commitlog/CommitLog-600-1630582314923.log, with bad position but valid CRC
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.shouldSkipSegmentOnError(CommitLogReplayer.java:438)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.handleUnrecoverableError(CommitLogReplayer.java:452)
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:109)
    at org.apache.cassandra.db.commitlog.CommitLogSegmentReader$SegmentIterator.computeNext(CommitLogSegmentReader.java:84)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readCommitLogSegment(CommitLogReader.java:236)
    at org.apache.cassandra.db.commitlog.CommitLogReader.readAllFiles(CommitLogReader.java:134)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.replayFiles(CommitLogReplayer.java:154)
    at org.apache.cassandra.db.commitlog.CommitLog.recoverFiles(CommitLog.java:213)
    at org.apache.cassandra.db.commitlog.CommitLog.recoverSegmentsOnDisk(CommitLog.java:194)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:338)
    at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:527)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:702)
    at com.datastax.bdp.DseModule.main(DseModule.java:96)

根因分析 #

故障节点本地的CommitLog（提交日志）文件头部损坏，虽CRC校验有效，但日志段的位置信息异常，Cassandra启动时重放提交日志失败，触发JVM稳定性检查直接强制退出。 #

前置确认 #

• 集群配置的Keyspace副本数≥2，其余8个节点均处于正常运行状态，无数据丢失风险
• 故障Pod绑定的PV/PVC可正常读写，排除底层存储损坏、挂载权限异常问题
• 故障节点近期无强制断电、硬重启类操作

修复方案 #

方案1：跳过损坏日志段（低风险优先选择） #

• 修改故障Pod的启动命令为sleep 3600，临时启动Pod，避免Cassandra持续崩溃退出
• 进入Pod，进入/var/lib/cassandra/commitlog/目录，将损坏的日志文件CommitLog-600-1630582314923.log移动到备份目录留存，不要直接删除
• 修改Cassandra配置文件cassandra.yaml，添加配置commitlog.skip_corrupted: true，设置启动时自动跳过损坏的提交日志段
• 恢复Pod原启动配置，重启Pod，观察启动日志确认服务是否正常

方案2：重建故障节点（方案1无效时选择） #

Cassandra集群副本机制会自动同步数据，无丢失风险：

• 删除故障Pod对应的PVC，清理损坏的本地数据
• 删除故障Pod，Kubernetes会自动调度重建新的Pod并绑定新PVC
• 新Pod启动后会自动加入集群，从其他正常节点同步全量数据，同步完成后节点恢复正常

注意事项 #

• 所有操作前必须确认集群其余节点状态正常，所有Keyspace的副本同步无滞后
• 损坏的CommitLog文件建议留存备份，可用于后续定位损坏根因

内容的提问来源于stack exchange，提问作者Mohammad Abbas