如何通过GitHub Actions推送KFP Pipeline至VertexAI Registry？（认证报错）

代码实现 #

pipeline_func = time_series_pipeline
pipeline_filename = 'time_series_pipeline.yaml'

print("compiling pipeline...")
compiler.Compiler().compile(
    pipeline_func=pipeline_func, package_path=pipeline_filename
)

parser = argparse.ArgumentParser(description='GCP access token.')
parser.add_argument('access_token', type=str, help='GCP access token')
args = parser.parse_args()
access_token = args.access_token

creds, project = google.auth.default()
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)

impersenated_creds = google.auth.impersonated_credentials.Credentials(
    source_credentials=environ['GOOGLE_APPLICATION_CREDENTIALS'],
    target_principal='gh-action@[project].iam.gserviceaccount.com'
)

registry = RegistryClient(
    host='https://europe-west3-kfp.pkg.dev/project_id/vertex-pipeline-registry',
    # auth=impersenated_creds
)

print("uploading pipeline...")
templateName, versionName = registry.upload_pipeline(
  file_name=pipeline_filename,
  tags=["v1", "latest"],
  extra_headers={"description":"This is an example pipeline template."})

报错信息 #

模拟凭据时的错误 #

Traceback (most recent call last):
compiling pipeline...
  File "/home/runner/work/project/project/vertex-pipeline/pipeline.py", line 112, in <module>
    creds.refresh(auth_req)
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/google/auth/external_account.py", line 401, in refresh
    self._impersonated_credentials.refresh(request)
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/google/auth/impersonated_credentials.py", line 235, in refresh
    self._update_token(request)
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/google/auth/impersonated_credentials.py", line 267, in _update_token
    self.token, self.expiry = _make_iam_token_request(
                              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/google/auth/impersonated_credentials.py", line 83, in _make_iam_token_request
    raise exceptions.RefreshError(_REFRESH_ERROR, response_body)
google.auth.exceptions.RefreshError: ('Unable to acquire impersonated credentials', '{
  "error": {
    "code": 400,
    "message": "Request contains an invalid argument.",
    "status": "INVALID_ARGUMENT"
  }
}
')

使用ApiAuth时的401错误 #

File "/home/runner/work/project/project/vertex-pipeline/pipeline.py", line 114, in <module>
    templateName, versionName = registry.upload_pipeline(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/kfp/registry/registry_client.py", line 352, in upload_pipeline
    response.raise_for_status()
  File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://europe-west3-kfp.pkg.dev/***/vertex-pipeline-registry

服务账号权限 #

gh-action服务账号已配置以下权限：

• AI Platform Admin
• Artifact Registry Administrator
• BigQuery Admin
• BigQuery Data Editor
• Cloud Functions Developer
• Cloud Functions Invoker
• Cloud Run Invoker
• Editor
• Eventarc Event Receiver
• Pub/Sub Admin
• Secret Manager Secret Accessor
• Service Account Token Creator
• Storage Admin
• Storage Object User
• Storage Transfer Admin
• Storage Transfer agent
• Storage Transfer Service service agent
• Vertex AI administrator

GitHub Actions工作流配置 #

env:
  PROJECT_ID: ${{ secrets.PROJECT_ID }}
  TF_BACKEND: ${{ secrets.TF_BACKEND }}
  TF_VAR_gh_token: ${{ secrets.TF_VAR_gh_token }}
  TF_IN_AUTOMATION: "true"

jobs:
  terraform-apply:
    runs-on: 'ubuntu-latest'
    permissions:
      contents: 'read'
      id-token: 'write'
      issues: 'write'
      pull-requests: 'write'

    steps:
      - name: Checkout repository
        uses: 'actions/checkout@v3'

      - id: 'auth'
        uses: 'google-github-actions/auth@v2'
        with:
          token_format: 'access_token'
          workload_identity_provider: ${{ secrets.WIF_PROVIDER_NAME }}
          service_account: ${{ secrets.SERVICE_ACCOUNT_EMAIL }}
      
      - uses: 'google-github-actions/setup-gcloud@v2'
        with:
          install_components: "beta,terraform-tools,gsutil,core"

      - uses: 'actions/setup-python@v5'
        if: steps.changes.outputs.vertex == 'true'
        with:
          python-version: 3.12

      - name: Install python dependencies
        if: steps.changes.outputs.vertex == 'true'
        run: |
          echo "Changed files in vertex-pipeline"
          python -m pip install --upgrade pip
          pip install -r vertex-pipeline/requirements.txt

      - name: Run vertex pipeline
        if: steps.changes.outputs.vertex == 'true'
        run: python vertex-pipeline/pipeline.py ${{steps.auth.outputs.access_token}}

已尝试的方案 #

• 将服务账号令牌传入RegistryClient()的auth参数
• 将凭据文件传入auth_file参数
• 模拟不同账号
• 使用ApiAuth()传入id_token和auth_token认证

所有尝试均失败，要么返回Unable to acquire impersonated credentials，要么返回401未授权。

1. 移除不必要的凭据模拟逻辑 #

GitHub Actions通过WIF认证后，环境已持有有效的服务账号凭据，无需手动模拟其他账号。原代码中用临时凭据（GOOGLE_APPLICATION_CREDENTIALS指向的WIF临时凭据）模拟目标账号的操作是错误的，临时凭据不支持该操作。

修改后的认证代码：

pipeline_func = time_series_pipeline
pipeline_filename = 'time_series_pipeline.yaml'

print("compiling pipeline...")
compiler.Compiler().compile(
    pipeline_func=pipeline_func, package_path=pipeline_filename
)

# 直接使用WIF配置的默认凭据
creds, project = google.auth.default()
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)

# 用默认凭据初始化RegistryClient，替换占位符为实际项目ID
registry = RegistryClient(
    host=f'https://europe-west3-kfp.pkg.dev/{project}/vertex-pipeline-registry',
    auth=creds
)

print("uploading pipeline...")
templateName, versionName = registry.upload_pipeline(
  file_name=pipeline_filename,
  tags=["v1", "latest"],
  extra_headers={"description":"This is an example pipeline template."})

2. 调整GitHub Actions认证配置 #

移除token_format: 'access_token'参数，不需要手动传递令牌，WIF会自动配置环境凭据：

- id: 'auth'
  uses: 'google-github-actions/auth@v2'
  with:
    workload_identity_provider: ${{ secrets.WIF_PROVIDER_NAME }}
    service_account: ${{ secrets.SERVICE_ACCOUNT_EMAIL }}

同时修改运行脚本的步骤，不再传递access token：

- name: Run vertex pipeline
  if: steps.changes.outputs.vertex == 'true'
  run: python vertex-pipeline/pipeline.py

3. 验证Artifact Registry权限 #

确认目标Artifact Registry（vertex-pipeline-registry）存在于europe-west3区域，并给服务账号绑定Artifact Registry Writer角色（比Administrator更精细，避免权限冗余）。

4. 增加凭据有效性验证 #

在GitHub Actions中添加验证步骤，确认WIF认证成功：

- name: Verify credentials
  run: gcloud auth print-access-token

内容的提问来源于stack exchange，提问作者chocho.boss