K8s集群中Python gRPC客户端与服务端通信问题排查及解决

在同一K8s集群的不同命名空间部署了Python gRPC客户端与服务端，已配置Ingress、Service和DNSService用于通信，但两者无法建立连接。要求使用非安全通道，不采用安全通道方案。

注：两者的K8s Service均将端口80映射到9000。

代码片段 #

客户端代码（client.py） #

GRPC_SERVER_ADDRESS = "grpc-server.com"

@image_bp.route('/image-grpc', methods=['POST'])
def modify_image_grpc():
    request_json = request.get_json()
    with grpc.insecure_channel(GRPC_SERVER_ADDRESS) as channel:
        stub = image_pb2_grpc.ImageServiceStub(channel)
        image = stub.GetImage(image_pb2.ImageRequest(
            id_user=request_json['id_user'],
        ))
    
    # response = ...

    return response

服务端代码（server.py） #

GRPC_MAX_WORKERS = 10
GRPC_HOST = "0.0.0.0"
GRPC_PORT = "9000"

def serve():
    try:
        server = grpc.server(futures.ThreadPoolExecutor(max_workers=GRPC_MAX_WORKERS))
        image_pb2_grpc.add_ImageServiceServicer_to_server(ImageService(), server)
        server.add_insecure_port(f'{GRPC_HOST}:{GRPC_PORT}')
        server.start()
        server.wait_for_termination()
    except Exception as e:
        logger.error(f'[-] Error serving gRPC on port {GRPC_PORT}: {e}')
        sys.exit()

报错信息 #

# client.error.log

<_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1662987231.739227750","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3260,"referenced_errors":[{"created":"@1662987231.739227058","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}"
>

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)

... (line in my code that raises the exception: stub.GetImage)

  File "/usr/local/lib64/python3.6/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/usr/local/lib64/python3.6/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses"
        debug_error_string = "{"created":"@1662987231.739227750","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3260,"referenced_errors":[{"created":"@1662987231.739227058","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":167,"grpc_status":14}]}"

依赖版本 #

# requirements.txt

...

grpcio==1.47.0
grpcio-tools==1.47.0

...

中间排查进展 #

直接使用服务端IP地址可成功连接，说明问题出在DNS名称解析或Ingress/Service配置上。

最终解决步骤（参考DazWilkin建议） #

通过以下操作实现了非安全通道的跨命名空间gRPC连接，无需依赖IP地址：

• 移除K8s Ingress，改用K8s集群内部的Service DNS名称，格式为{service-name}.{service-namespace}.svc.cluster.local
• 调整服务端K8s Service配置：将port和targetPort均设置为gRPC服务运行的端口9000（不再使用80端口映射）
• 更新客户端地址：将代码中的GRPC_SERVER_ADDRESS设置为<服务端Service DNS名称>:9000

内容的提问来源于stack exchange，提问作者Rodrigo Lima