Bitbucket流水线无法通过IAP连接GCP VM求助

背景 #

需要建立Google虚拟私有云的IAP隧道，访问未暴露公网的服务。官方命令为：

gcloud compute start-iap-tunnel vm-name target-port

正常执行会输出类似内容：

Picking local unused port [38441].
Testing if tunnel connection works.
Listening on port [38441].

其中"Listening on port ..."是关键成功标识。本地开发机用Bitbucket配置的服务账号凭证可正常运行，模拟流水线步骤的Docker容器也能成功执行，但在Bitbucket流水线中，命令会挂起，始终不输出该关键行。

Bitbucket流水线失败日志 #

启用详细日志后的执行记录：

gcloud compute start-iap-tunnel my-vm-name 8080 --zone=europe-west3-b --log-http --verbosity=debug
DEBUG: Running [gcloud.compute.start-iap-tunnel] with arguments: [--log-http: "true", --verbosity: "debug", --zone: "europe-west3-b", INSTANCE_NAME: "my-vm-name", INSTANCE_PORT: "8080"]
=======================
==== request start ====
uri: https://compute.googleapis.com/compute/v1/projects/<SANITIZED>/zones/europe-west3-b/instances/my-vm-name?alt=json
method: GET
== headers start ==
b'accept': b'application/json'
b'accept-encoding': b'gzip, deflate'
b'authorization': --- Token Redacted ---
b'content-length': b'0'
b'user-agent': b'google-cloud-sdk gcloud/436.0.0 command/gcloud.compute.start-iap-tunnel invocation-id/e86258106fc74f53b220deb1a2567db0 environment/None environment-version/None client-os/LINUX client-os-ver/5.15.0 client-pltf-arch/x86_64 interactive/False from-script/False python/3.9.16 term/ (Linux 5.15.0-1037-aws)'
b'x-goog-api-client': b'cred-type/sa'
== headers end ==
== body start ==

== body end ==
==== request end ====
DEBUG: Starting new HTTPS connection (1): compute.googleapis.com:443
DEBUG: https://compute.googleapis.com:443 "GET /compute/v1/projects/<SANITIZED>/zones/europe-west3-b/instances/my-vm-name?alt=json HTTP/1.1" 200 None
---- response start ----
status: 200
-- headers start --
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Encoding: gzip
Content-Type: application/json; charset=UTF-8
Date: Thu, 22 Jun 2023 16:05:56 GMT
ETag: <SANITIZED>
Server: ESF
Transfer-Encoding: chunked
Vary: Origin, X-Origin, Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 0
-- headers end --
-- body start --
{
"kind": "compute#instance",
"id": "<SANITIZED>",
"creationTimestamp": "2023-02-08T08:44:58.596-08:00",
"name": "my-vm-name",
"description": "",
"tags": {
"items": [
"<SANITIZED>",
"<SANITIZED>"
],
"fingerprint": "<SANITIZED>"
},
"machineType": "<SANITIZED>",
"status": "RUNNING",
"zone": "<SANITIZED>",
"canIpForward": false,
"networkInterfaces": [
{
"kind": "compute#networkInterface",
"network": "https://www.googleapis.com/compute/v1/projects/<SANITIZED>/global/networks/default",
"subnetwork": "https://www.googleapis.com/compute/v1/projects/<SANITIZED>/regions/europe-west3/subnetworks/default",
"networkIP": "10.156.0.27",
"name": "nic0",
"accessConfigs": [
{
"kind": "compute#accessConfig",
"type": "ONE_TO_ONE_NAT",
"name": "External NAT",
"natIP": "34.159.245.120",
"networkTier": "PREMIUM"
}
],
"fingerprint": "<SANITIZED>",
"stackType": "IPV4_ONLY"
}
],
"disks": [
<SANITIZED>
],
"metadata": {
"kind": "compute#metadata",
"fingerprint": "<SANITIZED>",
"items": [
{
"key": "windows-keys",
"value": "{<SANITIZED>}"
}
]
},
"serviceAccounts": [
{
"email": "<SANITIZED>@developer.gserviceaccount.com",
"scopes": [
"https://www.googleapis.com/auth/devstorage.read_only",
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring.write",
"https://www.googleapis.com/auth/servicecontrol",
"https://www.googleapis.com/auth/service.management.readonly",
"https://www.googleapis.com/auth/trace.append"
]
}
],
"selfLink": "https://www.googleapis.com/compute/v1/projects/<SANITIZED>/zones/europe-west3-b/instances/my-vm-name",
"scheduling": {
"onHostMaintenance": "TERMINATE",
"automaticRestart": true,
"preemptible": false,
"provisioningModel": "STANDARD"
},
"cpuPlatform": "Intel Cascade Lake",
"labelFingerprint": "<SANITIZED>",
"startRestricted": false,
"deletionProtection": false,
"reservationAffinity": {
"consumeReservationType": "ANY_RESERVATION"
},
"displayDevice": {
"enableDisplay": false
},
"shieldedInstanceConfig": {
"enableSecureBoot": false,
"enableVtpm": true,
"enableIntegrityMonitoring": true
},
"shieldedInstanceIntegrityPolicy": {
"updateAutoLearnPolicy": true
},
"confidentialInstanceConfig": {
"enableConfidentialCompute": false
},
"fingerprint": "<SANITIZED>",
"lastStartTimestamp": "2023-06-05T06:32:48.873-07:00",
"lastStopTimestamp": "2023-06-05T06:20:24.134-07:00",
"keyRevocationActionType": "NONE"
}

-- body end --
total round trip time (request+response): 0.211 secs
---- response end ----
----------------------
Picking local unused port [36513].
WARNING:

To increase the performance of the tunnel, consider installing NumPy. For instructions,
please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidth

Testing if tunnel connection works.
DEBUG: credentials type for _GetAccessTokenCallback is [<google.oauth2.service_account.Credentials object at 0x7fadb593edf0>].
DEBUG: Using new websocket library
INFO: Connecting with URL ['wss://tunnel.cloudproxy.app/v4/connect?project=<SANITIZED>&port=8080&newWebsocket=True&zone=europe-west3-b&instance=my-vm-name&interface=nic0']
DEBUG: RECV opcode [2] data_len [348] binary_data[:20] [b'\x00\x01\x00\x00\x01VAbvJZZ7uNrVf1j']
DEBUG: CLOSE
INFO: Received WebSocket Close message [None: 'Connection closed while receiving data.']. 

本地成功执行的最后几行日志 #

Testing if tunnel connection works.
DEBUG: credentials type for _GetAccessTokenCallback is [<google.oauth2.service_account.Credentials object at 0x7f58cb680a30>].
DEBUG: Using new websocket library
INFO: Connecting with URL ['wss://tunnel.cloudproxy.app/v4/connect?project=lofty-seer-161814&port=8080&newWebsocket=True&zone=europe-west3-b&instance=my-vm-name&interface=nic0']
DEBUG: RECV opcode [2] data_len [348] binary_data[:20] [b'\x00\x01\x00\x00\x01VAbvJZZ5Kz16cIn']
DEBUG: CLOSE
Listening on port [37863].
DEBUG: CLOSE
INFO: Received WebSocket Close message [None: 'Connection closed while receiving data.']. 

排查建议 #

• 检查Bitbucket流水线环境的网络出站规则：确认是否允许wss://tunnel.cloudproxy.app的WebSocket流量，CI/CD环境常存在此类连接限制
• 验证服务账号权限：确保流水线使用的服务账号拥有roles/iap.tunnelResourceAccessor角色，以及对目标VM的compute.instances.get权限
• 尝试指定固定本地端口：添加--local-port参数指定固定端口，避免自动端口分配时的潜在问题
• 更新gcloud SDK版本：流水线中使用的gcloud版本为436.0.0，升级至最新版本，可能包含WebSocket连接相关的bug修复
• 检查VM防火墙规则：确认VM所在网络配置了允许IAP隧道流量的规则（允许来自35.235.240.0/20的TCP流量到目标端口8080）
• 检查流水线资源限制：确认Bitbucket流水线的CPU、内存配额足够，资源不足可能导致隧道进程初始化停滞
• 尝试添加--no-check-certificate参数：跳过证书检查，测试是否是CI环境证书链问题导致连接中断

内容的提问来源于stack exchange，提问作者Anton Daneyko