GCP Dataproc集群置备超时 无法连接googleapis.com问题求助

我正在GCP项目中创建采用默认值的基础配置Dataproc集群，虚拟机可正常创建，但集群始终处于置备状态直至超时。

• 我先后通过控制台和命令行两种方式尝试创建集群
• 我测试了多个镜像版本（2.0-debian、2.0-ubuntu、1.5-debian、1.5-ubuntu）
• 未选择额外组件，集群将用于运行Spark作业

上述所有尝试均出现相同问题，我通过SSH登录集群主节点，在/var/log/google-dataproc-agent.0.log日志中查看到如下报错：
Network is unreachable: dataproccontrol-europe-west1.googleapis.com/2a00:1450:400c:c04:0:0:0:5f:443

完整错误栈如下：

Jul 24, 2021 11:02:53 AM com.google.cloud.hadoop.services.repackaged.com.google.cloud.hadoop.util.ResilientOperation nextSleep INFO: Transient exception caught. Sleeping for 1120, then retrying.
com.google.cloud.hadoop.services.repackaged.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 29.974635818s. [buffered_nanos=30006131805, waiting_for_connection]
        at com.google.cloud.hadoop.services.repackaged.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:244)
        at com.google.cloud.hadoop.services.repackaged.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:225)
        at com.google.cloud.hadoop.services.repackaged.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:142)
        at com.google.cloud.dataproc.control.v1.AgentServiceGrpc$AgentServiceBlockingStub.createAgent(AgentServiceGrpc.java:735)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater$1.call(AgentApiAsyncUpdater.java:238)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater$1.call(AgentApiAsyncUpdater.java:235)
        at com.google.cloud.hadoop.services.repackaged.com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:67)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.executeWithBackoff(AgentApiAsyncUpdater.java:345)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.createAgent(AgentApiAsyncUpdater.java:234)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.getOrCreateAgent(AgentApiAsyncUpdater.java:203)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.run(AgentApiAsyncUpdater.java:183)
        at com.google.cloud.hadoop.services.repackaged.com.google.common.util.concurrent.MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.run(MoreExecutors.java:679)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Jul 24, 2021 11:03:23 AM com.google.cloud.hadoop.services.repackaged.com.google.cloud.hadoop.util.ResilientOperation nextSleep INFO: Transient exception caught. Sleeping for 1958, then retrying.
com.google.cloud.hadoop.services.repackaged.io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at com.google.cloud.hadoop.services.repackaged.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:244)
        at com.google.cloud.hadoop.services.repackaged.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:225)
        at com.google.cloud.hadoop.services.repackaged.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:142)
        at com.google.cloud.dataproc.control.v1.AgentServiceGrpc$AgentServiceBlockingStub.createAgent(AgentServiceGrpc.java:735)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater$1.call(AgentApiAsyncUpdater.java:238)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater$1.call(AgentApiAsyncUpdater.java:235)
        at com.google.cloud.hadoop.services.repackaged.com.google.cloud.hadoop.util.ResilientOperation.retry(ResilientOperation.java:67)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.executeWithBackoff(AgentApiAsyncUpdater.java:345)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.createAgent(AgentApiAsyncUpdater.java:234)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.getOrCreateAgent(AgentApiAsyncUpdater.java:203)
        at com.google.cloud.hadoop.services.agent.protocol.AgentApiAsyncUpdater.run(AgentApiAsyncUpdater.java:183)
        at com.google.cloud.hadoop.services.repackaged.com.google.common.util.concurrent.MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.run(MoreExecutors.java:679)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.cloud.hadoop.services.repackaged.io.netty.channel.AbstractChannel$AnnotatedSocketException: Network is unreachable: dataproccontrol-europe-west1.googleapis.com/2a00:1450:400c:c04:0:0:0:5f:443
Caused by: java.net.SocketException: Network is unreachable
        at sun.nio.ch.Net.connect0(Native Method)
        at sun.nio.ch.Net.connect(Net.java:482)
        at sun.nio.ch.Net.connect(Net.java:474)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:91)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:88)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.internal.SocketUtils.connect(SocketUtils.java:88)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:315)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:248)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:533)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:54)
        at com.google.cloud.hadoop.services.repackaged.io.grpc.netty.WriteBufferingAndExceptionHandler.connect(WriteBufferingAndExceptionHandler.java:150)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:548)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.AbstractChannelHandlerContext.access$1000(AbstractChannelHandlerContext.java:61)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.AbstractChannelHandlerContext$9.run(AbstractChannelHandlerContext.java:538)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at com.google.cloud.hadoop.services.repackaged.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at com.google.cloud.hadoop.services.repackaged.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)

补充说明 #

我的防火墙与VPC配置如下：

集群配置如下：

问题排查与解决方案 #

报错核心原因是集群节点尝试通过IPv6地址访问Dataproc控制平面API失败，可按以下步骤排查解决：

1. 强制禁用IPv6访问Dataproc控制平面
   GCP默认VPC/子网未启用IPv6配置时，节点无法通过IPv6访问公网API，创建集群时强制指定使用IPv4即可解决：
   • 命令行创建时添加参数：--gce-private-ipv6-access=disabled
   • 控制台创建时，在「虚拟机配置」→「网络」分类下，关闭「私有IPv6访问」选项
2. 验证VPC网络访问能力
   • 如果集群部署在无公网IP的私有子网，需要为对应子网开启「专用Google访问通道」：进入VPC网络→子网列表，选中集群使用的子网，编辑勾选「启用专用 Google 访问通道」后保存
   • 如果集群节点分配了公网IP，确认防火墙出站规则允许443端口的TCP流量，从你提供的配置截图看出站规则默认允许，无需额外调整
3. 手动验证连通性
   SSH登录集群主节点执行以下命令，确认网络连通正常：

   # 验证IPv4解析正常
   nslookup dataproccontrol-europe-west1.googleapis.com
   # 验证API可达
   curl -I https://dataproccontrol-europe-west1.googleapis.com
   

   命令执行正常的话重新创建集群即可完成置备。

内容的提问来源于stack exchange，提问作者med