Riak集群节点移除失败：分区移交受阻与数据恢复咨询

运行5节点Riak KV 2.9.10集群，所有节点版本一致。为升级磁盘执行节点移除操作时，目标节点在迁移最后一个partition到集群内另一节点时停滞，接收节点因不可恢复错误崩溃，导致停滞节点上Riak服务自动关闭。

控制台错误日志 #

2024-06-08 07:35:26.278 [error] <0.801.0>@riak_kv_vnode:init:856 Failed to start riak_kv_eleveldb_backend backend for index 959110449498405040071168171470060731649205731328 error: {db_open,"Corruption: truncated record at end of file"}
2024-06-08 07:35:26.283 [notice] <0.801.0>@riak:stop:43 "backend module failed to start."
2024-06-08 07:35:26.283 [error] <0.801.0> gen_fsm <0.801.0> in state started terminated with reason: no function clause matching riak_kv_vnode:terminate({bad_return_value,{stop,{db_open,"Corruption: truncated record at end of file"}}}, undefined) line 2380
2024-06-08 07:35:26.283 [error] <0.801.0> CRASH REPORT Process <0.801.0> with 1 neighbours exited with reason: no function clause matching riak_kv_vnode:terminate({bad_return_value,{stop,{db_open,"Corruption: truncated record at end of file"}}}, undefined) line 2380 in gen_fsm:terminate/7 line 600
2024-06-08 07:35:26.283 [error] <0.170.0> Supervisor riak_core_vnode_sup had child undefined started with {riak_core_vnode,start_link,undefined} at <0.801.0> exit with reason no function clause matching riak_kv_vnode:terminate({bad_return_value,{stop,{db_open,"Corruption: truncated record at end of file"}}}, undefined) line 2380 in context child_terminated
2024-06-08 07:35:26.284 [error] <0.167.0> Supervisor riak_core_sup had child riak_core_vnode_manager started with riak_core_vnode_manager:start_link() at <0.215.0> exit with reason {{function_clause,[{riak_kv_vnode,terminate,[{bad_return_value,{stop,{db_open,"Corruption: truncated record at end of file"}}},undefined],[{file,"src/riak_kv_vnode.erl"},{line,2380}]},{riak_core_vnode,terminate,3,[{file,"src/riak_core_vnode.erl"},{line,941}]},{gen_fsm,terminate,7,[{file,"gen_fsm.erl"},{line,597}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},{gen_fsm,sync_send_event,[<0.801.0>,wait_for_init,infinity]}} in context child_terminated
2024-06-08 07:35:26.285 [info] <0.385.0>@riak_kv_app:prep_stop:267 Stopping application riak_kv - marked service down.

LevelDB分区日志 #

2024/06/08-07:35:26.275337 7f98e97e2640                        Version: 2.0.36 (enterprise edition)
2024/06/08-07:35:26.275362 7f98e97e2640             Options.comparator: leveldb.InternalKeyComparator
2024/06/08-07:35:26.275365 7f98e97e2640      Options.create_if_missing: 1
2024/06/08-07:35:26.275369 7f98e97e2640        Options.error_if_exists: 0
2024/06/08-07:35:26.275372 7f98e97e2640        Options.paranoid_checks: 0
2024/06/08-07:35:26.275375 7f98e97e2640     Options.verify_compactions: 1
2024/06/08-07:35:26.275378 7f98e97e2640                    Options.env: 0x7f99700049d0
2024/06/08-07:35:26.275382 7f98e97e2640               Options.info_log: 0x7f991c003260
2024/06/08-07:35:26.275385 7f98e97e2640      Options.write_buffer_size: 43049366
2024/06/08-07:35:26.275388 7f98e97e2640         Options.max_open_files: 1000
2024/06/08-07:35:26.275392 7f98e97e2640            Options.block_cache: 0x7f991c002070
2024/06/08-07:35:26.275395 7f98e97e2640             Options.block_size: 4096
2024/06/08-07:35:26.275398 7f98e97e2640       Options.block_size_steps: 16
2024/06/08-07:35:26.275401 7f98e97e2640 Options.block_restart_interval: 16
2024/06/08-07:35:26.275405 7f98e97e2640            Options.compression: 2
2024/06/08-07:35:26.275408 7f98e97e2640          Options.filter_policy: leveldb.BuiltinBloomFilter2
2024/06/08-07:35:26.275411 7f98e97e2640              Options.is_repair: false
2024/06/08-07:35:26.275414 7f98e97e2640         Options.is_internal_db: false
2024/06/08-07:35:26.275417 7f98e97e2640      Options.total_leveldb_mem: 11518763827
2024/06/08-07:35:26.275421 7f98e97e2640  Options.block_cache_threshold: 33554432
2024/06/08-07:35:26.275424 7f98e97e2640  Options.limited_developer_mem: false
2024/06/08-07:35:26.275427 7f98e97e2640              Options.mmap_size: 0
2024/06/08-07:35:26.275430 7f98e97e2640       Options.delete_threshold: 1000
2024/06/08-07:35:26.275434 7f98e97e2640       Options.fadvise_willneed: false
2024/06/08-07:35:26.275437 7f98e97e2640      Options.tiered_slow_level: 0
2024/06/08-07:35:26.275440 7f98e97e2640     Options.tiered_fast_prefix: /var/lib/riak/leveldb/959110449498405040071168171470060731649205731328
2024/06/08-07:35:26.275443 7f98e97e2640     Options.tiered_slow_prefix: /var/lib/riak/leveldb/959110449498405040071168171470060731649205731328
2024/06/08-07:35:26.275446 7f98e97e2640                         crc32c: hardware
2024/06/08-07:35:26.275449 7f98e97e2640   Options.cache_object_warming: true
2024/06/08-07:35:26.275453 7f98e97e2640        Options.ExpiryActivated: false
2024/06/08-07:35:26.275456 7f98e97e2640  ExpiryModuleEE.expiry_enabled: false
2024/06/08-07:35:26.275459 7f98e97e2640  ExpiryModuleEE.expiry_minutes: 0
2024/06/08-07:35:26.275462 7f98e97e2640     ExpiryModuleEE.whole_files: true
2024/06/08-07:35:26.275465 7f98e97e2640                File cache size: 1418935620
2024/06/08-07:35:26.275468 7f98e97e2640               Block cache size: 1421032772
2024/06/08-07:35:26.278268 7f98e97e2640 File cache warmed with 0 files.
2024/06/08-07:35:26.278306 7f98e97e2640 Wrote 0 file cache objects for warming.

已尝试修复分区但未成功，现考虑将故障节点标记为下线并替换新的空节点，需确保新节点通过Riak复制机制获取缺失数据。

解决方案 #

1. 标记故障节点永久下线 #

在集群任意正常节点执行以下命令，彻底移除故障节点：

riak-admin cluster leave <故障节点名称>
riak-admin cluster plan
riak-admin cluster commit

执行后验证节点状态：

riak-admin member-status

2. 添加新节点到集群 #

• 在新节点安装同版本Riak KV 2.9.10，确保riak.conf配置与集群其他节点一致（集群名称、节点标识、网络参数等）。
• 启动新节点后，执行加入集群命令：

riak-admin cluster join <正常节点名称>

• 回到任意正常节点，提交集群变更：

riak-admin cluster plan
riak-admin cluster commit

• 查看partition迁移进度：

riak-admin transfers

3. 触发数据修复保障完整性 #

由于原partition迁移未完成，部分数据副本可能缺失，需手动触发修复：

• 全局修复：修复所有bucket数据，确保副本数符合配置的n_val：

riak-admin repair

• 指定partition修复：针对日志中损坏的partition单独修复：

riak-admin repair partition 959110449498405040071168171470060731649205731328

4. 验证数据与集群状态 #

• 确认新节点完全加入集群：

riak-admin member-status

• 检查partition分配是否均衡、副本数达标：

riak-admin ring-status

• 随机读取关键bucket的key，验证数据一致性与可用性。

关键注意事项 #

• Riak的最终一致性模型会自动同步数据，但手动修复可加速副本补全，避免数据不可用。
• 新节点配置必须与集群完全一致，尤其是n_val、存储后端参数，否则会导致同步失败。
• 修复过程耗时与数据量、网络带宽相关，期间尽量减少大规模写入操作。

内容的提问来源于stack exchange，提问作者user103373