Terraform GSuite提供程序能否在Google Cloud Build中访问Directory API?
我决定使用Terraform实现GCP项目创建流程的自动化。运行Terraform时会创建全新的GSuite用户,这部分操作通过GSuite对应的Terraform provider完成。我已经完成了全部前置配置,包括Service Account配置、全域权限委托(domain-wide delegation)等,本地命令行执行Terraform的全流程运行一切正常。
之后我不再依赖本地命令行,打算配置Cloud Build触发器自动执行Terraform的init、plan、apply流程。Cloud Build默认的运行身份是内置的Cloud Build Service Account,我已经为该SA配齐了Terraform执行需要的所有GCP侧权限,但实际运行构建时发现,Terraform唯一无法成功创建的资源就是GSuite用户。
排查日志时找到了两个关键的API请求及响应:
GET /admin/directory/v1/users?alt=json&customer=my_customer&prettyPrint=false&query=email%3Alolloso-admin%40codedby.pm HTTP/1.1 Host: www.googleapis.com User-Agent: google-api-go-client/0.5 (linux amd64) Terraform/0.14.7 X-Goog-Api-Client: gl-go/1.15.6 gdcl/20200514 Accept-Encoding: gzip HTTP/2.0 400 Bad Request Cache-Control: private Content-Type: application/json; charset=UTF-8 Date: Sun, 28 Feb 2021 12:58:25 GMT Server: ESF Vary: Origin Vary: X-Origin Vary: Referer X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN X-Xss-Protection: 0 { "error": { "code": 400, "message": "Invalid Input", "errors": [ { "domain": "global", "reason": "invalid" } ] } }
POST /admin/directory/v1/users?alt=json&prettyPrint=false HTTP/1.1 Host: www.googleapis.com User-Agent: google-api-go-client/0.5 (linux amd64) Terraform/0.14.7 Content-Length: 276 Content-Type: application/json X-Goog-Api-Client: gl-go/1.15.6 gdcl/20200514 Accept-Encoding: gzip { "changePasswordAtNextLogin": true, "externalIds": [], "includeInGlobalAddressList": true, "name": { "familyName": "********", "givenName": "*******" }, "orgUnitPath": "/", "password": "********", "primaryEmail": "*********", "sshPublicKeys": [] } HTTP/2.0 403 Forbidden Cache-Control: private Content-Type: application/json; charset=UTF-8 Date: Sun, 28 Feb 2021 12:58:25 GMT Server: ESF Vary: Origin Vary: X-Origin Vary: Referer Www-Authenticate: Bearer realm="https://accounts.google.com/", error="insufficient_scope", scope="https://www.googleapis.com/auth/admin.directory.user https://www.googleapis.com/auth/directory.user" X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN X-Xss-Protection: 0 { "error": { "code": 403, "message": "Request had insufficient authentication scopes.", "errors": [ { "message": "Insufficient Permission", "domain": "global", "reason": "insufficientPermissions" } ], "status": "PERMISSION_DENIED" } }
从API返回的报错判断,问题根源是Cloud Build的Service Account没有Directory API的足够权限,但后续排查过程并不顺利:
- 最初尝试给内置的Cloud Build SA配置全域权限委托,但这类内置SA没有对应的配置入口,无法完成设置。
- 之后尝试在已经配置好全域权限委托的自有SA上,给Cloud Build SA绑定
serviceAccountUser角色,构建依然返回相同的权限不足报错,没有效果。 - 再之后尝试把已经配置好全域权限委托的自有SA设为自定义的Cloud Build运行身份,问题依然没有解决。
核心疑问:Cloud Build本身是不是无法访问需要通过全域权限委托才能操作的GSuite资源?
第一次更新:使用自定义构建Service Account的验证结果
按照建议尝试使用用户指定的Service Account执行构建,完全按照官方配置步骤操作,最终的cloudbuild.yaml配置如下:
steps: - id: 'tf init' name: 'hashicorp/terraform' entrypoint: 'sh' args: - '-c' - | terraform init - id: 'tf plan' name: 'hashicorp/terraform' entrypoint: 'sh' args: - '-c' - | terraform plan - id: 'tf apply' name: 'hashicorp/terraform' entrypoint: 'sh' args: - '-c' - | terraform apply -auto-approve logsBucket: 'gs://tf-project-creator-cloudbuild-logs' serviceAccount: 'projects/tf-project-creator/serviceAccounts/sa-terraform-project-creator@tf-project-creator.iam.gserviceaccount.com' options: env: - 'TF_LOG=DEBUG'
其中sa-terraform-project-creator@tf-project-creator.iam.gserviceaccount.com是已经在Google Workspace后台完成全域权限委托配置的Service Account。手动触发构建时执行的命令是:
export GOOGLE_APPLICATION_CREDENTIALS=.secrets/sa-terraform-project-creator.json; gcloud builds submit --config cloudbuild.yaml
执行时指定了该SA的JSON私钥,原本预期构建可以顺利通过,但依然返回了相同的权限报错:
POST /admin/directory/v1/users?alt=json&prettyPrint=false HTTP/1.1 Host: www.googleapis.com User-Agent: google-api-go-client/0.5 (linux amd64) Terraform/0.14.7 Content-Length: 276 Content-Type: application/json X-Goog-Api-Client: gl-go/1.15.6 gdcl/20200514 Accept-Encoding: gzip { "changePasswordAtNextLogin": true, "externalIds": [], "includeInGlobalAddressList": true, "name": { "familyName": "REDACTED", "givenName": "REDACTED" }, "orgUnitPath": "/", "organizations": [], "password": "REDACTED", "primaryEmail": "REDACTED", "sshPublicKeys": [] } ----------------------------------------------------- 2021/03/06 17:26:19 [DEBUG] Google API Response Details: ---[ RESPONSE ]-------------------------------------- HTTP/2.0 403 Forbidden Cache-Control: private Content-Type: application/json; charset=UTF-8 Date: Sat, 06 Mar 2021 17:26:19 GMT Server: ESF Vary: Origin Vary: X-Origin Vary: Referer Www-Authenticate: Bearer realm="https://accounts.google.com/", www.googleapis.com/auth/directory.user" X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN X-Xss-Protection: 0 { "error": { "code": 403, "message": "Request had insufficient authentication scopes.", "errors": [ { "message": "Insufficient Permission", "domain": "global", "reason": "insufficientPermissions" } ], "status": "PERMISSION_DENIED" } }
此时不确定是否还有遗漏的配置项。
第二次更新:核查构建提交时的实际生效身份
针对之前的验证结果,又做了两项排查:
- 在Cloud Build设置页启用"Service Accounts"选项,和预期一样没有改善。
- 再次核查提交构建时的实际生效身份:使用自定义构建SA的场景下,必须手动触发构建,之前执行
gcloud builds submit --config cloudbuild.yaml命令时,总会先设置GOOGLE_APPLICATION_CREDENTIALS环境变量:
指定的私钥属于已经配置好全域权限委托的构建SA。之前执行该操作时,gcloud登录的账号是另一个项目Owner账号,这个账号没有全域权限委托权限。原本的判断是设置了export GOOGLE_APPLICATION_CREDENTIALS=.secrets/sa-terraform-project-creator.jsonGOOGLE_APPLICATION_CREDENTIALS环境变量后,gcloud会自动使用该变量指定的凭据。但后续尝试直接用构建SA登录gcloud再提交构建,操作命令如下:
完成认证后立即执行构建提交命令,访问Directory API时依然遇到相同的权限问题。gcloud auth activate-service-account sa-terraform-project-creator@tf-project-creator.iam.gserviceaccount.com --key-file='.secrets/sa-terraform-project-creator.json'
现在怀疑的方向是:构建执行过程中,调用Directory API时用了错误的凭据。当前需要找到方法打印Terraform插件调用API时使用的身份日志,以此定位问题。
内容的提问来源于stack exchange,提问作者coder.pm

