GCP Secret Manager突发Bad Gateway错误排查求助

我开发的Cloud Function每3分钟连接Secret Manager获取凭证对接数据库，稳定运行数月，但近2小时未做任何变更的情况下，频繁出现未见过的gRPC相关错误。

错误日志 #

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 75, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/grpc/_channel.py", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/grpc/_channel.py", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "502:Bad Gateway"
    debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"502:Bad Gateway", grpc_status:14, created_time:"2023-11-08T18:19:37.079022828+00:00"}"
>
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/retry.py", line 204, in retry_target
return target()
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
return func(*args, **kwargs)
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 77, in error_remapped_callable
raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.ServiceUnavailable: 503 502:Bad Gateway

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/workspace/main.py", line 45, in handler
result = get_message(event, context, temp_flow_id)
File "/workspace/main.py", line 228, in get_message
write_bqcache(bqcache_query, mode_test)
File "/workspace/main.py", line 245, in write_bqcache
BQCACHE_LOGIN, BQCACHE_PASSWORD, BQCACHE_IP = ssktools.get_authentication("BQCACHE_LOGIN_PASSWORD_IP").split(
File "/workspace/ssktools.py", line 108, in get_authentication
return __access_secret_version(PROJECT_ID, secret_id, "latest")
File "/workspace/ssktools.py", line 80, in __access_secret_version
response = client.access_secret_version(request={"name": name})
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1518, in access_secret_version
response = rpc(
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 131, in call
return wrapped_func(*args, **kwargs)
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/retry.py", line 366, in retry_wrapped_func
return retry_target(
File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/retry.py", line 220, in retry_target
raise exceptions.RetryError(
google.api_core.exceptions.RetryError: Deadline of 60.0s exceeded while calling target function, last exception: 503 502:Bad Gateway

密钥获取实现代码 #

def __access_secret_version(project_id: str, secret_id: str, version_id: str) -> str:
"""
Access the payload for the given secret version if one exists. The version
can be a version number as a string (e.g. "5") or an alias (e.g. "latest").

Args:
    project_id (str): 项目ID
    secret_id (str): 目标密钥ID
    version_id (str): 版本号

Returns:
    string: 密钥内容
"""
# Create the Secret Manager client.
client = secretmanager.SecretManagerServiceClient()

# Build the resource name of the secret version.
name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"

# Access the secret version.
response = client.access_secret_version(request={"name": name})

payload = response.payload.data.decode("UTF-8")  # type: ignore

return payload

原因分析及修复方案 #

• 排查GCP服务状态：这类502/503错误大概率和平台侧临时波动有关，优先确认Secret Manager或对应区域的GCP网络服务是否存在故障。
• 复用Secret Manager客户端：当前代码每次调用都新建客户端实例，频繁创建gRPC连接易触发限流或连接不稳定。改为全局初始化客户端，复用连接：

  # 全局初始化客户端，避免每次函数调用重复创建
  client = secretmanager.SecretManagerServiceClient()
  
  def __access_secret_version(project_id: str, secret_id: str, version_id: str) -> str:
      """
      Access the payload for the given secret version if one exists. The version
      can be a version number as a string (e.g. "5") or an alias (e.g. "latest").
  
      Args:
          project_id (str): 项目ID
          secret_id (str): 目标密钥ID
          version_id (str): 版本号
  
      Returns:
          string: 密钥内容
      """
      name = f"projects/{project_id}/secrets/{secret_id}/versions/{version_id}"
      response = client.access_secret_version(request={"name": name})
      payload = response.payload.data.decode("UTF-8")  # type: ignore
      return payload
  

• 自定义重试策略：默认重试规则可能未覆盖这类502场景，增加针对服务不可用、超时错误的重试次数和间隔：

  from google.api_core import retry
  from google.api_core.exceptions import ServiceUnavailable, DeadlineExceeded
  
  # 自定义重试规则：针对503/超时错误，最长重试120秒，间隔从1秒翻倍到10秒
  custom_retry = retry.Retry(
      retry.if_exception_type(ServiceUnavailable, DeadlineExceeded),
      initial=1.0,
      total=120.0,
      multiplier=2,
      maximum=10.0,
  )
  
  # 调用时传入自定义重试策略
  response = client.access_secret_version(request={"name": name}, retry=custom_retry)
  

• 检查网络配置：如果Cloud Function使用VPC连接器，确认VPC防火墙规则是否限制了Secret Manager的访问，或连接器本身是否存在异常。

内容的提问来源于stack exchange，提问作者Yassine