Envoy Sidecar未监听端口致Consul健康检查失败求助

我们在Nomad集群的每个工作负载服务旁部署了Envoy Docker sidecar，本应接入Consul服务网格。工作负载服务已正确部署并可在Nomad网络访问，但Consul无法通过Envoy sidecar的健康检查：

Connect Sidecar Listening
ServiceName my-service-sidecar-proxy
CheckID service:_nomad-task-xxxxxx-xxxx-xxxxxx-xxxxx-xxxxxxxxxx-group-my-service-my-service-interface-sidecar-proxy:2
Type tcp
Output dial tcp 10.0.0.72:30241: connect: connection refused

从服务器可通过私有网络或Nomad网络访问工作负载服务，但无法访问Envoy：

# Workload service
server-node-0:~$ wget 10.0.0.63:28947/health
--2022-09-07 15:38:28--  http://10.0.0.63:28947/health
Connecting to 10.0.0.63:28947... connected.
HTTP request sent, awaiting response... 200 OK
# Envoy
server-node-0:~$ wget 10.0.0.63:30241
--2022-09-07 15:38:57--  http://10.0.0.63:30241/
Connecting to 10.0.0.63:30241... failed: Connection refused.

在服务部署节点或Docker容器内执行netstat，仅工作负载服务端口开放，Envoy端口未开放：

client-node-1:~$ netstat -a | grep 28947
tcp        0      0 client-node-1:49236 client-node-1:28947 TIME_WAIT
client-node-1:~$ netstat -a | grep 30241
client-node-1:~$

但Envoy进程在sidecar容器中运行正常，其管理端口19001可正常响应：

root@3d7c60d8b94a:/# ps auxww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
envoy        1  0.1  1.3 2256608 53300 ?       Ssl  15:53   0:05 envoy -c /secrets/envoy_bootstrap.json -l debug --concurrency 1 --disable-hot-restart
root@3d7c60d8b94a:/# curl -X POST 127.0.0.2:19001/ready
LIVE

开启调试日志后，发现以下关键信息：

[2022-09-07 17:03:13.173][1][debug][config] [./source/common/config/grpc_stream.h:61] Establishing new gRPC bidi stream for rpc DeltaAggregatedResources(stream .envoy.service.discovery.v3.DeltaDiscoveryRequest) returns (stream .envoy.service.discovery.v3.DeltaDiscoveryResponse);

[2022-09-07 17:03:13.173][1][debug][router] [source/common/router/router.cc:471] [C0][S3688902394289443788] cluster 'local_agent' match for URL '/envoy.service.discovery.v3.AggregatedDiscoveryService/DeltaAggregatedResources'
[2022-09-07 17:03:13.173][1][debug][router] [source/common/router/router.cc:675] [C0][S3688902394289443788] router decoding headers:
':method', 'POST'
':path', '/envoy.service.discovery.v3.AggregatedDiscoveryService/DeltaAggregatedResources'
':authority', 'local_agent'
':scheme', 'http'
'te', 'trailers'
'content-type', 'application/grpc'
'x-consul-token', 'bf88ad2f-7d22-716c-6e5f-8c404f866d6e'
'x-envoy-internal', 'true'
'x-forwarded-for', '172.26.64.4'

[2022-09-07 17:03:13.173][1][debug][pool] [source/common/http/conn_pool_base.cc:78] queueing stream due to no available connections (ready=0 busy=0 connecting=0)
[2022-09-07 17:03:13.173][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:290] trying to create new connection
[2022-09-07 17:03:13.173][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:145] creating a new connection (connecting=0)
[2022-09-07 17:03:13.173][1][debug][http2] [source/common/http/http2/codec_impl.cc:1788] [C294] updating connection-level initial window size to 268435456
[2022-09-07 17:03:13.173][1][debug][connection] [./source/common/network/connection_impl.h:89] [C294] current connecting state: true
[2022-09-07 17:03:13.173][1][debug][client] [source/common/http/codec_client.cc:57] [C294] connecting
[2022-09-07 17:03:13.173][1][debug][connection] [source/common/network/connection_impl.cc:924] [C294] connecting to alloc/tmp/consul_grpc.sock
[2022-09-07 17:03:13.174][1][debug][connection] [source/common/network/connection_impl.cc:683] [C294] connected
[2022-09-07 17:03:13.174][1][debug][client] [source/common/http/codec_client.cc:89] [C294] connected
[2022-09-07 17:03:13.174][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:327] [C294] attaching to next stream
[2022-09-07 17:03:13.174][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:181] [C294] creating stream
[2022-09-07 17:03:13.174][1][debug][router] [source/common/router/upstream_request.cc:424] [C0][S3688902394289443788] pool ready
[2022-09-07 17:03:13.179][1][debug][connection] [source/common/network/connection_impl.cc:651] [C294] remote close
[2022-09-07 17:03:13.179][1][debug][connection] [source/common/network/connection_impl.cc:250] [C294] closing socket: 0
[2022-09-07 17:03:13.179][1][debug][client] [source/common/http/codec_client.cc:108] [C294] disconnect. resetting 1 pending requests
[2022-09-07 17:03:13.179][1][debug][client] [source/common/http/codec_client.cc:140] [C294] request reset
[2022-09-07 17:03:13.179][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:214] [C294] destroying stream: 0 remaining
[2022-09-07 17:03:13.179][1][debug][router] [source/common/router/router.cc:1198] [C0][S3688902394289443788] upstream reset: reset reason: connection termination, transport failure reason:
[2022-09-07 17:03:13.179][1][debug][http] [source/common/http/async_client_impl.cc:101] async http request response headers (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection termination'

[2022-09-07 17:03:13.179][1][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 4161s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination
[2022-09-07 17:03:13.179][1][debug][config] [source/common/config/grpc_subscription_impl.cc:113] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
[2022-09-07 17:03:13.179][1][debug][config] [source/common/config/grpc_subscription_impl.cc:113] gRPC update for type.googleapis.com/envoy.config.listener.v3.Listener failed
[2022-09-07 17:03:13.179][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:483] [C294] client disconnected, failure reason:
[2022-09-07 17:03:13.179][1][debug][pool] [source/common/conn_pool/conn_pool_base.cc:453] invoking idle callbacks - is_draining_for_deletion_=false

Envoy版本：1.23.0
Consul版本：1.13.1

问题原因及解决办法 #

• Consul gRPC套接字挂载异常
  日志显示Envoy连接alloc/tmp/consul_grpc.sock后立即被关闭，说明sidecar容器未正确挂载Consul Agent的gRPC Unix套接字，或套接字权限配置错误，导致Envoy无法与Consul Agent建立稳定gRPC连接，无法获取监听器配置，因此不会监听代理端口。
  解决：检查Nomad任务定义，确认Consul Agent的gRPC套接字路径（通常为/consul/grpc/consul_grpc.sock）已正确挂载到sidecar容器的alloc/tmp/consul_grpc.sock；同时确保套接字文件权限允许Envoy用户（默认uid 101）读写。

• Consul Agent gRPC服务未启用
  若Consul Agent未开启gRPC服务，即使套接字存在，Envoy也无法通过它获取配置。
  解决：检查Consul Agent配置，确保grpc_enabled = true，且grpc_port（默认8502）配置正确；确认Agent进程中gRPC服务正常运行。

• Envoy引导配置错误
  Envoy进程启动正常，但引导配置可能存在错误，导致无法正确连接Consul的xDS服务、获取监听器配置。比如local_agent集群地址配置错误，或Consul token权限不足，无法获取服务网格配置。
  解决：检查/secrets/envoy_bootstrap.json中的xDS配置，确认Consul地址、token权限正确；使用grpcurl工具连接套接字测试Consul Agent的gRPC服务可用性。

• 网络命名空间隔离问题
  Nomad任务的网络配置可能导致sidecar容器无法访问Consul Agent的套接字，或监听器端口未正确暴露到Nomad网络中。
  解决：检查Nomad任务的network块配置，确保sidecar容器端口已正确映射，且网络模式允许容器访问主机上的Consul套接字（如使用host网络或正确挂载）。

内容的提问来源于stack exchange，提问作者Bob