Kubernetes部署的Bitbucket自托管Runner中Docker Buildx权限问题求助

问题概述 #

在AWS EKS环境中部署的Bitbucket自托管Runner执行Docker Buildx构建多平台镜像时，出现授权错误，提示authorization denied by plugin pipelines: --privileged=true is not allowed。该命令在本地和VM环境的Runner中运行正常，仅在EKS上触发问题。

环境配置 #

Bitbucket Pipeline配置 #

pipelines:
  branches:
    master:
      - step:
          name: Docker build
          runs-on:
            - 'self.hosted'
            - 'linux'
          services:
            - docker
          script:
            - echo "Executing on self-hosted runner" 
            - echo "$DOCKERHUB_PASSWORD" | docker login --username "$DOCKERHUB_USERNAME" --password-stdin
            - wget https://github.com/docker/buildx/releases/download/v0.18.0/buildx-v0.18.0.linux-amd64
            - chmod +x buildx-v0.18.0.linux-amd64
            - mv buildx-v0.18.0.linux-amd64 docker-buildx
            - mkdir -p ~/.docker/cli-plugins
            - mv docker-buildx ~/.docker/cli-plugins
            - docker buildx build -t gurgenyegoryan/test-bitbucket:latest --push --platform linux/arm64 . # Build multi-architecture image for the platforms provided and push to Dockerhub

Kubernetes Runner Job配置 #

apiVersion: v1
kind: List
items:
  - apiVersion: v1
    kind: Secret
    metadata:
      name: runner-oauth-credentials
      labels:
        accountUuid: *************************
        repositoryUuid: ***********************
        runnerUuid: ****************************
    data:
      oauthClientId: *****************************=
      oauthClientSecret: *********************************************=
  
  - apiVersion: batch/v1
    kind: Job
    metadata:
      name: runner
    spec:
      template:
        metadata:
          labels:
            accountUuid: *************************
            repositoryUuid: ***********************
            runnerUuid: ****************************
        spec:
          containers:
            - name: runner
              image: docker-public.packages.atlassian.com/sox/atlassian/bitbucket-pipelines-runner
              securityContext:
                privileged: true
              env:
                - name: DOCKER_BUILDKIT
                  value: 0
                - name: ACCOUNT_UUID
                  value: "{*****************}"
                - name: REPOSITORY_UUID
                  value: "{*************************-826239d933d2}"
                - name: RUNNER_UUID
                  value: "{**********************}"
                - name: OAUTH_CLIENT_ID
                  valueFrom:
                    secretKeyRef:
                      name: runner-oauth-credentials
                      key: oauthClientId
                - name: OAUTH_CLIENT_SECRET
                  valueFrom:
                    secretKeyRef:
                      name: runner-oauth-credentials
                      key: oauthClientSecret
                - name: WORKING_DIRECTORY
                  value: "/tmp"
                - name: RUNTIME_PREREQUISITES_ENABLED
                  value: "true"
              volumeMounts:
                - name: tmp
                  mountPath: /tmp
                - name: docker-containers
                  mountPath: /var/lib/docker/containers
                  readOnly: true
                - name: var-run
                  mountPath: /var/run
            - name: docker-in-docker
              image: docker:20.10.7-dind
              env:
                - name: DOCKER_BUILDKIT
                  value: "0"
              securityContext:
                privileged: true
              volumeMounts:
                - name: tmp
                  mountPath: /tmp
                - name: docker-containers
                  mountPath: /var/lib/docker/containers
                - name: var-run
                  mountPath: /var/run
          restartPolicy: OnFailure
          volumes:
            - name: tmp
            - name: docker-containers
            - name: var-run

报错信息 #

docker buildx build -t gurgenyegoryan/test-bitbucket:latest --push --platform linux/arm64 . #0 building with "default" instance using docker-container driver #1 [internal] booting buildkit #1 pulling image moby/buildkit:buildx-stable-1 #1 pulling image moby/buildkit:buildx-stable-1 6.0s done #1 creating container buildx_buildkit_default 0.0s done #1 ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed ------ > [internal] booting buildkit: ------ ERROR: Error response from daemon: authorization denied by plugin pipelines: --privileged=true is not allowed

解决方案 #

方法1：手动启动Buildkit实例并让Buildx连接（推荐） #

Bitbucket Runner的Docker插件默认阻止创建privileged容器，而Buildx自动启动的buildkit容器需要该权限。可以手动在dind容器中启动buildkit，再让Buildx连接到该实例：

修改Pipeline的script部分：

- echo "Executing on self-hosted runner" 
- echo "$DOCKERHUB_PASSWORD" | docker login --username "$DOCKERHUB_USERNAME" --password-stdin
# 手动启动privileged的Buildkit容器
- docker run -d --name buildkitd --privileged moby/buildkit:buildx-stable-1
# 创建Buildx实例并绑定到已启动的Buildkit
- docker buildx create --use --name custom_builder --driver-opt network=host
# 执行多平台构建推送
- docker buildx build -t gurgenyegoryan/test-bitbucket:latest --push --platform linux/arm64 .
# 清理资源（可选）
- docker buildx rm custom_builder
- docker stop buildkitd && docker rm buildkitd

方法2：修改Runner配置允许Privileged容器 #

在Kubernetes Job的runner容器环境变量中添加允许创建privileged容器的配置：

- name: DOCKER_ALLOW_PRIVILEGED
  value: "true"

添加后重新部署Runner Job，Buildx将能自动启动buildkit容器。

方法3：检查EKS PodSecurityPolicy（PSP） #

如果EKS集群启用了PodSecurityPolicy，需确保Runner使用的ServiceAccount被授权使用允许privileged的PSP：

1. 创建允许privileged的PSP：

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: runner-privileged
spec:
  privileged: true
  allowPrivilegeEscalation: true
  allowedCapabilities: ["*"]
  volumes: ["*"]
  runAsUser:
    rule: "RunAsAny"
  seLinux:
    rule: "RunAsAny"
  supplementalGroups:
    rule: "RunAsAny"
  fsGroup:
    rule: "RunAsAny"

2. 创建ClusterRole绑定该PSP：

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: psp-runner-privileged
rules:
- apiGroups: ["policy"]
  resources: ["podsecuritypolicies"]
  verbs: ["use"]
  resourceNames: ["runner-privileged"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: runner-psp-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: psp-runner-privileged
subjects:
- kind: ServiceAccount
  name: default # 替换为Runner使用的ServiceAccount名称
  namespace: your-namespace # 替换为Runner所在命名空间

方法4：启用Docker Buildkit（可选） #

将Kubernetes Job中DOCKER_BUILDKIT的值改为1，启用Buildkit可能解决部分权限兼容性问题：

- name: DOCKER_BUILDKIT
  value: "1"

内容的提问来源于stack exchange，提问作者Gurgen Yegoryan