Hyperledger Fabric Orderer从v2.4.4升级至v2.5后无响应问题求助

问题场景 #

5节点Orderer集群从v2.4.4升级至v2.5前已完成通道端点配置更新，但升级后所有Orderer均无法响应请求，集群丧失法定人数。该问题可稳定复现：更新通道配置块端点后执行Orderer升级即触发异常。

异常日志 #

2023-05-22 22:12:15.626 UTC 0036 INFO [orderer.consensus.etcdraft] confirmSuspicion -> Suspecting our own eviction from the channel for 10m0.03452979s channel=publicchannel node=6
2023-05-22 22:12:15.655 UTC 0037 INFO [orderer.common.cluster.puller] fetchLastBlockSeq -> orderer01.domain.xyz:7050 is at block sequence of 34 channel=publicchannel
2023-05-22 22:12:15.655 UTC 0038 INFO [orderer.common.cluster.puller] fetchLastBlockSeq -> orderer02.domain.xyz:7050 is at block sequence of 34 channel=publicchannel
2023-05-22 22:12:15.656 UTC 0039 INFO [orderer.common.cluster.puller] fetchLastBlockSeq -> orderer03.domain.xyz:7050 is at block sequence of 34 channel=publicchannel
2023-05-22 22:12:15.657 UTC 003a INFO [orderer.common.cluster.puller] fetchLastBlockSeq -> orderer00.domain.xyz:7050 is at block sequence of 34 channel=publicchannel
2023-05-22 22:12:15.657 UTC 003b INFO [orderer.common.cluster.puller] fetchLastBlockSeq -> orderer04.domain.xyz:7050 is at block sequence of 34 channel=publicchannel
2023-05-22 22:12:15.657 UTC 003c INFO [orderer.common.cluster.puller] HeightsByEndpoints -> Returning the heights of OSNs mapped by endpoints map[orderer00.domain.xyz:7050:35 orderer01.domain.xyz:7050:35 orderer02.domain.xyz:7050:35 orderer03.domain.xyz:7050:35 orderer04.domain.xyz:7050:35] channel=publicchannel
2023-05-22 22:12:15.658 UTC 003d INFO [orderer.consensus.etcdraft] confirmSuspicion -> Last config block was found to be block [34] channel=publicchannel node=6
2023-05-22 22:12:15.659 UTC 003e INFO [comm.grpc.server] 1 -> streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=10.60.22.184:49650 grpc.peer_subject="CN=orderer00.domain.xyz,OU=orderer,O=Hyperledger,ST=North Carolina,C=US" error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=9.036094ms
2023-05-22 22:12:15.661 UTC 003f INFO [orderer.consensus.etcdraft] confirmSuspicion -> Cannot confirm our own eviction from the channel, our certificate was found in config block with sequence 34 channel=publicchannel node=6
2023-05-22 22:12:16.069 UTC 0040 INFO [comm.grpc.server] 1 -> streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=10.60.27.163:60880 grpc.peer_subject="CN=orderer02.domain.xyz,OU=orderer,O=Hyperledger,ST=North Carolina,C=US" error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=9.513544ms
2023-05-22 22:12:16.705 UTC 0041 INFO [comm.grpc.server] 1 -> streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=10.60.27.163:60884 grpc.peer_subject="CN=orderer01.domain.xyz,OU=orderer,O=Hyperledger,ST=North Carolina,C=US" error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=13.161555ms
2023-05-22 22:12:17.412 UTC 0042 INFO [comm.grpc.server] 1 -> streaming call completed grpc.service=orderer.AtomicBroadcast grpc.method=Deliver grpc.peer_address=10.60.27.163:60900 grpc.peer_subject="CN=orderer03.domain.xyz,OU=orderer,O=Hyperledger,ST=North Carolina,C=US" error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=10.398404ms

排查与解决思路 #

• 校验节点ID与通道配置一致性
  日志中出现node=6，但集群仅包含5个节点，需确认每个Orderer的ETCDRAFT_NODE_ID环境变量或配置文件中的节点ID，是否与通道配置块（block 34）内定义的etcdraft节点ID完全匹配。v2.5版本对节点ID的校验逻辑更严格，ID不匹配会触发节点自我驱逐怀疑，导致集群无法形成法定人数。

• 确认通道端点配置更新的完整性
  使用configtxlator工具导出并检查配置块（block 34）的内容，确认所有5个Orderer的新端点已正确写入配置，且每个Orderer均已同步到该配置块：

  configtxlator proto_decode --input config_block.pb --type common.Block | jq .data.data[0].payload.data.config
  

  若存在端点配置缺失或同步不完整，需重新更新通道配置并确保所有节点完成同步后再执行升级。

• 检查etcdraft共识参数兼容性
  Fabric v2.5对etcdraft模块的超时参数（如TickInterval、ElectionTick、HeartbeatTick）做了优化，若升级后参数配置与v2.4.4时期不一致，可能导致节点间心跳超时、无法建立共识连接。需确认所有Orderer的共识配置参数统一，且符合v2.5版本的推荐值。

• 排查TLS证书与gRPC连接问题
  日志中多次出现context finished before block retrieved: context canceled，大概率是节点间gRPC连接建立失败。需检查：

  • 每个Orderer的TLS证书是否包含新端点的SAN（Subject Alternative Name）字段；
  • 节点间网络连通性，确认7050端口可正常访问；
  • 证书信任链是否完整，所有Orderer的根证书一致。

• 采用滚动升级策略
  避免全量升级所有Orderer，改为滚动升级：先升级一个节点，确认其正常加入集群、同步块数据后，再依次升级剩余节点。全量重启可能导致集群重新选举时出现配置不一致，触发法定人数丧失。

• 重置etcdraft本地数据（备份后操作）
  若节点本地etcdraft数据存在缓存不一致情况，可备份每个Orderer的/var/hyperledger/production/orderer/etcdraft目录后删除，重启节点让其重新同步通道配置与块数据，修复本地数据不一致问题。

内容的提问来源于stack exchange，提问作者Akshay Sood