Hyperledger Fabric跨Org节点连接及共识异常问题排查求助

问题背景 #

搭建了包含3个ORG、1个通道的Hyperledger Fabric网络，每个ORG各有1个Peer节点和1个Orderer节点。提交交易时，各ORG的Peer仅能连接本ORG的Orderer，尝试连接其他ORG的Orderer均失败；此外，更新Org1和Org2的证书后，Orderer2无法参与共识。

交易连接失败日志 #

WARN [gateway] orderers -> Failed to connect to orderer address=orderer.org2... err="failed to create new connection: connection error: desc = \"transport: error while dialing: dial tcp ...: connect: connection refused\""
WARN [gateway] orderers -> Failed to connect to orderer address=orderer.org3... err="failed to create new connection: connection error: desc = \"transport: error while dialing: dial tcp ...: connect: connection refused\""
INFO [gateway] Submit -> Sending transaction to orderer txID=... endpoint=orderer.org1.

通道Orderer配置 #

Consenters配置 #

"ConsensusType": {
  "mod_policy": "Admins",
  "value": {
    "metadata": {
      "consenters": [
        {
          "client_tls_cert": "updated cert",
          "host": "orderer.org1.xxx.com",
          "port": 7050,
          "server_tls_cert": "updated cert"
        },
        {
          "client_tls_cert": "updated cert",
          "host": "orderer.org2.x.com",
          "port": 8050,
          "server_tls_cert": "updated cert"
        },
        {
          "client_tls_cert": "old cert",
          "host": "orderer.org3.x.com",
          "port": 9050,
          "server_tls_cert": "old cert"
        }
      ],
      "options": {
        "election_tick": 10,
        "heartbeat_tick": 1,
        "max_inflight_blocks": 5,
        "snapshot_interval_size": 16777216,
        "tick_interval": "500ms"
      }
    }
  }
}

OrdererAddresses配置 #

"OrdererAddresses": {
  "mod_policy": "/Channel/Orderer/Admins",
  "value": {
    "addresses": [
      "orderer.org1.x.com:7050",
      "orderer.org2.x.com:8050",
      "orderer.org3.x.com:9050"
    ]
  },
  "version": "1"
}

Orderer2共识异常日志 #

3-10-15 23:30:38.272 UTC 0089 INFO [orderer.consensus.etcdraft] Step -> 3 [logterm: 14, index: 94826, vote: 3] ignored MsgPreVote from 2 [logterm: 14, index: 94823] at term 14: lease is not expired (remaining ticks: 2) channel=channel1 node=3
2023-10-15 23:30:44.269 UTC 008a INFO [orderer.consensus.etcdraft] Step -> 3 [logterm: 14, index: 94826, vote: 3] ignored MsgPreVote from 2 [logterm: 14, index: 94823] at term 14: lease is not expired (remaining ticks: 10) channel=channel1 node=3
3-10-15 23:30:50.269 UTC 008b INFO [orderer.consensus.etcdraft] Step -> 3 [logterm: 14, index: 94826, vote: 3] ignored MsgPreVote from 2 [logterm: 14, index: 94823] at term 14: lease is not expired (remaining ticks: 8) channel=channel1 node=3
3-10-15 23:30:56.269 UTC 008c INFO [orderer.consensus.etcdraft] Step -> 3 [logterm: 14, index: 94826, vote: 3] ignored MsgPreVote from 2 [logterm: 14, index: 94823] at term 14: lease is not expired (remaining ticks: 6) channel=channel1 node=3
3-10-15 23:31:02.269 UTC 008d INFO [orderer.consensus.etcdraft] Step -> 3 [logterm: 14, index: 94826, vote: 3] ignored MsgPreVote from 2 [logterm: 14, index: 94823] at term 14: lease is not expired (remaining ticks: 4) channel=channel1 node=3
3-10-15 23:31:08.269 UTC 008e INFO [orderer.consensus.etcdraft] Step -> 3 [logterm: 14, index: 94826, vote: 3] ignored MsgPreVote from 2 [logterm: 14, index: 94823] at term 14: lease is not expired (remaining ticks: 2) channel=channel1 node=3

Orderer3启动日志 #

3-10-13 18:50:11.971 UTC 0011 INFO [orderer.commmon.multichannel] initSystemChannel -> Starting system channel 'system-channel' with genesis block hash c9e6204d3e77735704a77cf3055be51997a4abe452e7d9b575caf0f023a82325 and orderer type etcdraft
2023-10-13 18:50:11.985 UTC 0012 INFO [orderer.consensus.etcdraft] HandleChain -> EvictionSuspicion not set, defaulting to 10m0s
2023-10-13 18:50:11.985 UTC 0013 INFO [orderer.consensus.etcdraft] HandleChain -> With system channel: after eviction InactiveChainRegistry.TrackChain will be called
2023-10-13 18:50:11.985 UTC 0014 INFO [orderer.consensus.etcdraft] createOrReadWAL -> Found WAL data at path '/var/hyperledger/production/orderer/etcdraft/wal/channel1', replaying it channel=channel1 node=3
2023-10-13 18:50:12.132 UTC 0015 INFO [orderer.consensus.etcdraft] Start -> Starting Raft node channel=system-channel node=3
2023-10-13 18:50:12.132 UTC 0016 INFO [orderer.common.cluster] Configure -> Entering, channel: system-channel, nodes: [ID: 1,
Endpoint: orderer.org1.xxx.com:7050,
ServerTLSCert:-----BEGIN CERTIFICATE-----
OLD CERT
-----END CERTIFICATE-----
, ClientTLSCert:-----BEGIN CERTIFICATE-----
OLD CERT
-----END CERTIFICATE-----
 ID: 2,
Endpoint: orderer.org2.xxx.com:7050,
ServerTLSCert:-----BEGIN CERTIFICATE-----
OLD CERT
-----END CERTIFICATE-----
, ClientTLSCert:-----BEGIN CERTIFICATE-----
OLD CERT
-----END CERTIFICATE-----
]
2023-10-13 18:50:12.132 UTC 0017 INFO [orderer.common.cluster] updateStubInMapping -> Allocating a new stub for node 1 with endpoint of orderer.org1.xxx.com:7050 for channel system-channel
2023-10-13 18:50:12.132 UTC 0018 INFO [orderer.common.cluster] updateStubInMapping -> Deactivating node 1 in channel system-channel with endpoint of orderer.org1.xxx.com:7050 due to TLS certificate change
2023-10-13 18:50:12.133 UTC 0019 INFO [orderer.common.cluster] updateStubInMapping -> Allocating a new stub for node 2 with endpoint of orderer.org2.xxx.com:7050 for channel system-channel
2023-10-13 18:50:12.133 UTC 001a INFO [orderer.common.cluster] updateStubInMapping -> Deactivating node 2 in channel system-channel with endpoint of orderer.org2.xxx.com:7050 due to TLS certificate change
2023-10-13 18:50:12.134 UTC 001b INFO [orderer.common.cluster] func1 -> 1 exists in both old and new membership for channel system-channel , skipping its deactivation
2023-10-13 18:50:12.134 UTC 001c INFO [orderer.common.cluster] func1 -> 2 exists in both old and new membership for channel system-channel , skipping its deactivation
2023-10-13 18:50:12.134 UTC 001d INFO [orderer.common.cluster] Configure -> Exiting
2023-10-13 18:50:12.134 UTC 001e INFO [orderer.consensus.etcdraft] start -> Restarting raft node channel=system-channel node=3
2023-10-13 18:50:12.135 UTC 001f INFO [orderer.consensus.etcdraft] becomeFollower -> 3 became follower at term 9 channel=system-channel node=3
2023-10-13 18:50:12.135 UTC 0020 INFO [orderer.consensus.etcdraft] newRaft -> newRaft 3 [peers: [], term: 9, commit: 11, applied: 0, lastindex: 12, lastterm: 9] channel=system-channel node=3
2023-10-13 18:50:12.135 UTC 0021 INFO [orderer.consensus.etcdraft] Start -> Starting Raft node channel=channel1 node=3
2023-10-13 18:50:12.135 UTC 0022 INFO [orderer.common.cluster] Configure -> Entering, channel: channel1, nodes: [ID: 1,
Endpoint: orderer.org1.xxx.com:7050,
ServerTLSCert:-----BEGIN CERTIFICATE-----
NEW CERT
-----END CERTIFICATE-----
, ClientTLSCert:-----BEGIN CERTIFICATE-----
NEW CERT
-----END CERTIFICATE-----
 ID: 2,
Endpoint: orderer.org2.xxx.com:8050,
ServerTLSCert:-----BEGIN CERTIFICATE-----
NEW CERT
-----END CERTIFICATE-----
, ClientTLSCert:-----BEGIN CERTIFICATE-----
NEW CERT
-----END CERTIFICATE-----
]
2023-10-13 18:50:12.135 UTC 0023 INFO [orderer.common.cluster] updateStubInMapping -> Allocating a new stub for node 1 with endpoint of orderer.org1.xxx.com:7050 for channel channel1
2023-10-13 18:50:12.135 UTC 0024 INFO [orderer.common.cluster] updateStubInMapping -> Deactivating node 1 in channel channel1 with endpoint of orderer.org1.xxx.com:7050 due to TLS certificate change
2023-10-13 18:50:12.135 UTC 0025 INFO [orderer.common.cluster] updateStubInMapping -> Allocating a new stub for node 2 with endpoint of orderer.org2.xxx.com:8050 for channel channel1
2023-10-13 18:50:12.135 UTC 0026 INFO [orderer.common.cluster] updateStubInMapping -> Deactivating node 2 in channel channel1 with endpoint of orderer.org2.xxx.com:8050 due to TLS certificate change
2023-10-13 18:50:12.136 UTC 0027 INFO [orderer.common.cluster] func1 -> 1 exists in both old and new membership for channel channel1 , skipping its deactivation
2023-10-13 18:50:12.136 UTC 0028 INFO [orderer.common.cluster] func1 -> 2 exists in both old and new membership for channel channel1 , skipping its deactivation
2023-10-13 18:50:12.136 UTC 0029 INFO [orderer.common.cluster] Configure -> Exiting
2023-10-13 18:50:12.136 UTC 002a INFO [orderer.consensus.etcdraft] start -> Restarting raft node channel=channel1 node=3
2023-10-13 18:50:12.136 UTC 002b INFO [orderer.consensus.etcdraft] becomeFollower -> 3 became follower at term 10 channel=channel1 node=3
2023-10-13 18:50:12.136 UTC 002c INFO [orderer.consensus.etcdraft] newRaft -> newRaft 3 [peers: [1,2,3], term: 10, commit: 94811, applied: 93447, lastindex: 94812, lastterm: 10] channel=channel1 node=3
2023-10-13 18:50:12.137 UTC 002d INFO [orderer.common.server] Main -> Starting orderer:
 Version: 2.4.9
 Commit SHA: 2a13518
 Go version: go1.18.10
 OS/Arch: linux/amd64
2023-10-13 18:50:12.137 UTC 002e INFO [orderer.common.server] Main -> Beginning to serve requests

问题分析与解决建议 #

1. Peer连接跨ORG Orderer失败问题 #

当前通道配置的OrdererAddresses包含了所有3个Orderer的地址，Peer会尝试轮询这些地址提交交易。如果网络中Peer无法访问其他ORG的Orderer（比如防火墙限制、网络隔离），就会出现连接拒绝的警告。

如果业务上只需要Peer连接本ORG的Orderer，可以调整通道的OrdererAddresses配置，只保留本ORG的Orderer地址。但注意：

• 若使用Raft共识，Orderer集群需要互相连通，但Peer仅需访问至少一个可用的Orderer节点即可提交交易
• 调整配置需要通过通道更新操作完成，由通道管理员执行

2. Orderer2无法参与共识问题 #

从日志看，Orderer3拒绝了Orderer2的预投票请求，原因是当前Leader的租约未到期，同时Orderer2的日志索引（94823）落后于Orderer3的日志索引（94826），说明Orderer2与集群同步异常。结合证书更新的背景，可能的原因：

• Orderer2的证书更新后，未正确重启或同步通道配置
• Orderer2的WAL日志损坏，导致无法追上集群的日志进度
• 网络连通性问题，Orderer2无法与其他Orderer节点同步数据

解决步骤：

1. 检查Orderer2与其他Orderer节点的网络连通性，确保端口可访问
2. 重启Orderer2，触发配置同步
3. 若同步仍失败，可尝试重置Orderer2的WAL日志（注意备份数据），重新加入集群
4. 确认通道配置中的Orderer2证书与实际使用的证书一致

内容的提问来源于stack exchange，提问作者Matias Salimbene