添加新DC后Cassandra nodetool rebuild因校验和不匹配失败

问题背景 #

在Ubuntu 20.04系统的COSS v4.1.3集群中新增数据中心后，执行nodetool rebuild时抛出java.io.IOException: Checksum didn't match异常，具体错误日志如下：

Error while reading partition DecoratedKey(-1908928440232805475, 000800fd592c2fa372470000014d00000a362a362a382a332a362a00) from stream on ks='xxxxxxxxxxx' and table='xxxxxxxxxxx'.
ERROR [Stream-Deserializer-/10.x.x.x:7002-ab4567ee] 2024-08-11 15:46:29,110 StreamSession.java:696 - [Stream #a3506520-17ca-11ef-8dcb-c97277257478] Streaming error occurred on session with peer 10.x.x.x:7002
java.io.IOException: Checksum didn't match (expected: 1541040037, actual: -1225830714)
        at org.apache.cassandra.db.streaming.CompressedInputStream.maybeValidateChecksum(CompressedInputStream.java:206)
        at org.apache.cassandra.db.streaming.CompressedInputStream.loadNextChunk(CompressedInputStream.java:157)
        at org.apache.cassandra.db.streaming.CompressedInputStream.reBuffer(CompressedInputStream.java:121)
        at org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:90)
        at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
        at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:62)
        at org.apache.cassandra.io.util.TrackedDataInputPlus.readFully(TrackedDataInputPlus.java:118)
        at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:433)
        at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:408)
        at org.apache.cassandra.db.streaming.CassandraStreamReader$StreamDeserializer.newPartition(CassandraStreamReader.java:211)
        at org.apache.cassandra.db.streaming.CassandraStreamReader.writePartition(CassandraStreamReader.java:184)
        at org.apache.cassandra.db.streaming.CassandraCompressedStreamReader.read(CassandraCompressedStreamReader.java:96)
        at org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:84)
        at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:50)
        at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36)
        at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
        at org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:59)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:750)
DEBUG [Stream-Deserializer-/10.x.x.x:7002-ab4567ee] 2024-08-11 15:46:29,111 StreamSession.java:551 - [Stream #a3506520-17ca-11ef-8dcb-c97277257478] Changing session state from STREAMING to FAILED
DEBUG [Stream-Deserializer-/10.x.x.x:7002-ab4567ee] 2024-08-11 15:46:29,111 StreamSession.java:551 - [Stream #a3506520-17ca-11ef-8dcb-c97277257478] Changing session state from FAILED to FAILED

解决方案 #

1. 检查并修复源节点数据完整性 #

• 在源DC的对应节点上，对报错的键空间和表执行数据校验：

  nodetool verify ks='xxxxxxxxxxx' table='xxxxxxxxxxx'
  

• 如果发现数据损坏，执行修复操作：

  nodetool repair ks='xxxxxxxxxxx' table='xxxxxxxxxxx'
  

• 修复完成后重新执行nodetool rebuild。

2. 临时禁用流数据压缩 #

该异常常因压缩数据流传输校验失败导致，可临时禁用流压缩绕过：

• 修改Cassandra配置文件cassandra.yaml，将stream_compression设置为none：

  stream_compression: none
  

• 重启新增DC的所有节点，再执行nodetool rebuild。
• 重建完成后，将stream_compression恢复为原配置（如lz4）并重启节点。

3. 排查网络传输问题 #

• 检查源节点与新增DC节点间的网络稳定性，使用工具测试丢包和延迟：

  mtr 10.x.x.x --report
  

• 若存在网络故障，先修复（如调整交换机端口、优化路由），再重试重建操作。

4. 手动指定源DC执行重建 #

执行nodetool rebuild时明确指定源数据中心，避免自动选节点出错：

nodetool rebuild --source-dc <源DC名称>

5. 清理临时流文件后重试 #

• 在新增DC节点上，清理Cassandra临时流文件目录（默认路径为/var/lib/cassandra/data/streaming）：

  rm -rf /var/lib/cassandra/data/streaming/*
  

• 重启节点后重新执行nodetool rebuild。

内容的提问来源于stack exchange，提问作者Vadivel