生产环境主节点Riak与Stanchion服务无法启动求助

问题现象 #

生产环境中Riak与Stanchion服务突然停止，控制台日志报错如下：

2023-11-07 03:41:14.547 [info] <0.160.0>@riak_core_app:stop:114 Stopped application riak_core.
2023-11-07 03:41:14.547 [error] <0.164.0> Supervisor riak_core_vnode_sup had child undefined started with riak_core_vnode:start_link() at undefined exit with reason killed in context shutdown_error
2023-11-07 03:41:14.547 [error] <0.164.0> Supervisor riak_core_vnode_sup had child undefined started with riak_core_vnode:start_link() at undefined exit with reason bad argument in call to ets:lookup(ets_riak_kv_entropy, {index,{riak_kv,1233142006497949337234359077604363797834693083136}}) in riak_kv_entropy_info:update_index_info/2 line 161 in context shutdown_error
2023-11-07 03:41:14.547 [error] <0.164.0> Supervisor riak_core_vnode_sup had child undefined started with riak_core_vnode:start_link() at undefined exit with reason bad argument in call to ets:lookup(ets_riak_kv_entropy, {index,{riak_kv,1187470080331358621040493926581979953470445191168}}) in riak_kv_entropy_info:update_index_info/2 line 161 in context shutdown_error

已尝试重启服务、删除ring目录文件并重新加入集群，问题未解决。

故障分析 #

日志核心错误为ets:lookup(ets_riak_kv_entropy, ...)参数异常，说明Riak熵计算模块依赖的ETS表（ets_riak_kv_entropy）损坏或未正确初始化，导致vnode进程启动失败，最终触发riak_core应用终止。

解决方案 #

1. 彻底清理熵相关数据 #

• 停止所有节点的Riak与Stanchion服务：

  sudo service riak stop
  sudo service stanchion stop
  

• 删除每个节点的熵数据目录（默认路径为/var/lib/riak/entropy，需根据实际配置调整）：

  rm -rf /var/lib/riak/entropy/*
  

2. 重置集群元数据缓存 #

• 清理ring目录及riak_core元数据：

  rm -rf /var/lib/riak/ring/*
  rm -rf /var/lib/riak/data/riak_core/*
  

3. 重新初始化集群 #

• 启动主节点并确认状态：

  sudo service riak start
  riak-admin status
  

• 启动其他节点并加入集群：

  sudo service riak start
  riak-admin cluster join riak@<主节点IP或主机名>
  riak-admin cluster plan
  riak-admin cluster commit
  

4. 验证服务恢复 #

• 检查集群节点状态：

  riak-admin member-status
  

• 启动Stanchion服务并验证：

  sudo service stanchion start
  stanchion-admin status
  

额外排查项 #

• 检查节点磁盘空间，避免因磁盘满导致数据写入失败：

  df -h
  

• 查看系统日志（/var/log/messages或/var/log/syslog），排查是否存在内存不足、磁盘IO错误等底层系统异常。

内容的提问来源于stack exchange，提问作者SujayW