GKE部署FastAPI应用调用Bigtable 30分钟后出现403权限拒绝问题

问题场景 #

我通过以下代码创建BigtableDataClient连接Bigtable实例并读取数据：

import logging
from os import path

from google.cloud.bigtable.data import BigtableDataClient
from google.auth.credentials import AnonymousCredentials
from ..config import config


FIRST = "first"
SECOND = "second"
THIRD = "third"

tables = {}

def initialize_bigtable_client():
    if path.isfile(config.BIGTABLE_SVCKEY):
        logging.info("Initializing bigtable client using credentials file")
        client = BigtableDataClient(
            project=config.BIGTABLE_PROJECT,
            client_options={
                "credentials_file": config.BIGTABLE_SVCKEY,
            },
        )
    else:
        logging.warning(f"Bigtable service account key not found at {config.BIGTABLE_SVCKEY}")
        credentials = AnonymousCredentials()
        client = BigtableDataClient(project=config.BIGTABLE_PROJECT, credentials=credentials)
    tables[FIRST] = client.get_table(config.BIGTABLE_INSTANCE, FIRST)
    tables[SECOND] = client.get_table(
        config.BIGTABLE_INSTANCE, SECOND
    )
    tables[THIRD] = client.get_table(config.BIGTABLE_INSTANCE, THIRD)
    return client

本地Docker运行FastAPI应用时可正常读取数据，但部署到GKE后，应用能正常运行约30分钟，之后每次调用都会触发错误：

google.api_core.exceptions.PermissionDenied: 403 Access denied. Missing IAM permission: bigtable.tables.readRows

本地与GKE使用相同的凭证文件，对应服务账号拥有Bigtable Reader角色。客户端在FastAPI启动事件中初始化，使用的bigtable库版本为google-cloud-bigtable==2.30.0。怀疑客户端丢失了创建时传入的凭证，转而使用环境变量GOOGLE_APPLICATION_CREDENTIALS中无对应权限的账号，但无法验证该猜想。

排查与解决方向 #

1. 显式加载凭证对象，避免默认逻辑覆盖 #

Google Cloud客户端库存在自动加载凭证的优先级机制，可能在凭证刷新时 fallback 到默认凭证。建议直接加载凭证对象传入客户端，确保始终使用指定的服务账号：

from google.oauth2 import service_account

def initialize_bigtable_client():
    if path.isfile(config.BIGTABLE_SVCKEY):
        logging.info("Initializing bigtable client using credentials file")
        # 显式加载凭证对象，指定必要权限范围
        credentials = service_account.Credentials.from_service_account_file(
            config.BIGTABLE_SVCKEY,
            scopes=["https://www.googleapis.com/auth/cloud-platform"]
        )
        client = BigtableDataClient(
            project=config.BIGTABLE_PROJECT,
            credentials=credentials
        )
        # 打印当前使用的服务账号，用于日志验证
        logging.info(f"Active service account: {credentials.service_account_email}")
    else:
        logging.warning(f"Bigtable service account key not found at {config.BIGTABLE_SVCKEY}")
        credentials = AnonymousCredentials()
        client = BigtableDataClient(project=config.BIGTABLE_PROJECT, credentials=credentials)
    # 保持原有表初始化逻辑
    tables[FIRST] = client.get_table(config.BIGTABLE_INSTANCE, FIRST)
    tables[SECOND] = client.get_table(config.BIGTABLE_INSTANCE, SECOND)
    tables[THIRD] = client.get_table(config.BIGTABLE_INSTANCE, THIRD)
    return client

2. 禁用GKE默认服务账号挂载 #

GKE会自动为Pod挂载节点的默认服务账号，可能在凭证刷新时被客户端库优先使用。可以在部署YAML中禁用自动挂载：

spec:
  serviceAccountName: default
  automountServiceAccountToken: false

3. 清理或固定环境变量 #

检查Pod内的GOOGLE_APPLICATION_CREDENTIALS环境变量，确保它指向正确的凭证文件，或者直接移除该变量，强制客户端使用代码中加载的凭证。

4. 监控凭证变化日志 #

通过添加的服务账号邮箱日志，对比GKE中应用运行初期和30分钟后的日志，确认是否出现了服务账号切换的情况，验证你的猜想。

内容的提问来源于stack exchange，提问作者Wojtek