ECS上Dagster Usercode Server访问失败问题排查求助

问题概述 #

基于原deploy_ecs示例适配（Docker停止支持ECS），用Terraform搭建ECS结构与ECR镜像仓库，修改docker-compose配置后，Usercode Server已正常运行，但Webserver无法通过AWS Cloud Map服务名dagster-usercode.dagster.local:4000访问它，报错提示DNS解析失败。

环境配置 #

ECS分布式部署架构 #

• 1台Webserver
• 1台Daemon Server
• 1台Usercode Server

Webserver的workspace.yaml配置 #

load_from:
  - python_file: job2.py
  - python_package:
      location_name: "webserver-jobs"
      package_name: job3
  - grpc_server:
      host: dagster-usercode.dagster.local
      port: 4000
      location_name: "usercode-jobs"

AWS Cloud Map配置 #

• 命名空间：dagster.local
• 服务dagster-usercode指向ECS服务实例dagster_usercode

AWS ECS状态 #

• 服务dagster_usercode已运行，对应ECR中的Usercode Server容器日志显示正常

Usercode Server启动命令 #

dagster api grpc -h 0.0.0.0 -p 4000 -f sample_jobs.py

加载workspace.yaml的现象 #

• job2可正常加载并调用
• job3报错（非重点，暂不关注）：

dagster._core.errors.DagsterInvariantViolationError: No repositories, jobs, pipelines, graphs, or asset definitions found in "job3".  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/server.py", line 408, in __init__    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/server.py", line 242, in __init__    loadable_targets = get_loadable_targets(  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/utils.py", line 60, in get_loadable_targets    else loadable_targets_from_python_package(package_name, working_directory)  File "/usr/local/lib/python3.10/site-packages/dagster/_core/workspace/autodiscovery.py", line 51, in loadable_targets_from_python_package    return loadable_targets_from_loaded_module(module)  File "/usr/local/lib/python3.10/site-packages/dagster/_core/workspace/autodiscovery.py", line 116, in loadable_targets_from_loaded_module    raise DagsterInvariantViolationError(

• gRPC连接报错（重点）：

dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE  File "/usr/local/lib/python3.10/site-packages/dagster/_core/workspace/context.py", line 614, in _load_location    else origin.create_location(self.instance)  File "/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/origin.py", line 373, in create_location    return GrpcServerCodeLocation(self, instance=instance)  File "/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/code_location.py", line 632, in __init__    list_repositories_response = sync_list_repositories_grpc(self.client)  File "/usr/local/lib/python3.10/site-packages/dagster/_api/list_repositories.py", line 20, in sync_list_repositories_grpc    api_client.list_repositories(),  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 250, in list_repositories    res = self._query("ListRepositories", api_pb2.ListRepositoriesRequest)  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 173, in _query    self._raise_grpc_exception(  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 156, in _raise_grpc_exception    raise DagsterUserCodeUnreachableError(The above exception was caused by the following exception:grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:   status = StatusCode.UNAVAILABLE details = "DNS resolution failed for dagster-usercode.dagster.local:4000: C-ares status is not ARES_SUCCESS qtype=A name=dagster-usercode.dagster.local is_balancer=0: Domain name not found"   debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-02-12T16:21:15.104749061+00:00", grpc_status:14, grpc_message:"DNS resolution failed for dagster-usercode.dagster.local:4000: C-ares status is not ARES_SUCCESS qtype=A name=dagster-usercode.dagster.local is_balancer=0: Domain name not found"}">  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 171, in _query    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 141, in _get_response    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 1160, in __call__    return _end_unary_response_blocking(state, call, False, None)  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking    raise _InactiveRpcError(state)  # pytype: disable=not-instantiableThe above exception occurred during handling of the following exception:dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/server_watcher.py", line 119, in watch_grpc_server_thread    watch_for_changes()  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/server_watcher.py", line 82, in watch_for_changes    new_server_id = client.get_server_id(timeout=REQUEST_TIMEOUT)  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 233, in get_server_id    res = self._query("GetServerId", api_pb2.Empty, timeout=timeout)  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 173, in _query    self._raise_grpc_exception(  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 156, in _raise_grpc_exception    raise DagsterUserCodeUnreachableError(The above exception was caused by the following exception:grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:   status = StatusCode.UNAVAILABLE details = "DNS resolution failed for dagster-usercode.dagster.local:4000: C-ares status is not ARES_SUCCESS qtype=AAAA name=dagster-usercode.dagster.local is_balancer=0: Domain name not found"    debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2024-02-12T16:21:04.309598593+00:00", grpc_status:14, grpc_message:"DNS resolution failed for dagster-usercode.dagster.local:4000: C-ares status is not ARES_SUCCESS qtype=AAAA name=dagster-usercode.dagster.local is_balancer=0: Domain name not found"}">  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 171, in _query    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)  File "/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py", line 141, in _get_response    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 1160, in __call__    return _end_unary_response_blocking(state, call, False, None)  File "/usr/local/lib/python3.10/site-packages/grpc/_channel.py", line 1003, in _end_unary_response_blocking    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable

下一步排查方向 #

1. 验证Cloud Map服务发现的DNS解析能力 #

• 进入Webserver所在ECS任务的容器，执行nslookup dagster-usercode.dagster.local或dig dagster-usercode.dagster.local，确认是否能解析到Usercode Server的私有IP。若解析失败，说明Cloud Map服务注册存在问题。
• 检查Cloud Map服务的实例列表：确认dagster-usercode服务下有健康的ECS实例注册，实例状态为正常。
• 确认Cloud Map命名空间类型：如果是私有DNS命名空间，需保证Webserver与Usercode Server在同一VPC，且VPC启用DNS支持，DHCP选项集配置了AWS提供的DNS服务器（通常为VPC网段+2，如10.0.0.2）。

2. 检查ECS任务的网络连通性配置 #

• 确认Webserver与Usercode Server的ECS任务是否在同一VPC、同一安全组，或安全组之间允许4000端口的TCP流量（gRPC基于TCP协议）。
• 查看Usercode Server任务的网络模式：若为awsvpc模式，确认任务分配的私有IP可被Webserver所在VPC访问。
• 验证Usercode Server的端口监听：通过容器日志确认服务已启动并监听0.0.0.0:4000（启动命令已配置-h 0.0.0.0，重点确认服务无启动异常）。

3. 绕过DNS直接测试gRPC连通性 #

• 临时修改Webserver的workspace.yaml，将host替换为Usercode Server的私有IP，测试是否能正常连通。若连通成功，说明问题确实在DNS解析；若仍失败，排查网络链路问题。
• 在Webserver容器内使用grpcurl工具测试：grpcurl -plaintext <usercode-ip>:4000 dagster.api.grpc.RepositoriesService/ListRepositories，确认是否能获取正常响应。

4. 检查Cloud Map与ECS的关联配置 #

• 确认ECS服务dagster_usercode的服务发现配置：是否正确关联了Cloud Map的dagster-usercode服务，且注册端口为4000。
• 检查Cloud Map的健康检查配置：若服务配置了健康检查，确认健康检查已通过，只有健康实例才会被DNS返回。

内容的提问来源于stack exchange，提问作者mjahr