Pod中Client-go创建K8s客户端间歇性报Unauthorized错误排查

客户在使用client-go创建Kubernetes Go客户端时遇到间歇性问题：客户端创建随机失败并返回**"Unauthorized"**错误。

示例代码片段 #

package main

import (
    "context"
    "fmt"
    
    corev1 "k8s.io/api/core/v1"
    extClient "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
    "k8s.io/apimachinery/pkg/api/meta"
    "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/types"
    utilruntime "k8s.io/apimachinery/pkg/util/runtime"
    client "k8s.io/client-go/kubernetes"
    clientGoScheme "k8s.io/client-go/kubernetes/scheme"
    "k8s.io/client-go/rest"
    ctrl "sigs.k8s.io/controller-runtime"
    ctrlRuntime "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/client/apiutil"
)

func main() {
    scheme := runtime.NewScheme()
    utilruntime.Must(clientGoScheme.AddToScheme(scheme))
    restConfig := ctrl.GetConfigOrDie()
    set, err := client.NewForConfig(restConfig)
    if err != nil {
        panic(err)
    }
    extSet, err := extClient.NewForConfig(restConfig)
    if err != nil {
        panic(err)
    }

    mapper, err := func(c *rest.Config) (meta.RESTMapper, error) {
        return apiutil.NewDynamicRESTMapper(c)
    }(restConfig)
    if err != nil {
        panic(err)
    }
    cl, err := ctrlRuntime.New(restConfig, ctrlRuntime.Options{
        Scheme: scheme,
        Mapper: mapper,
    })
    pod := &corev1.Pod{}
    err = cl.Get(context.Background(), types.NamespacedName{
        Namespace: "xyx",
        Name:      "abcd",
    }, pod)
    fmt.Println(err, set, extSet)
}

环境信息 #

• 发行版：OpenShift
• Kubernetes版本：1.21

怀疑方向与复现情况 #

目前怀疑是ServiceAccount token过期或损坏导致client-go创建客户端失败，已通过设置无效ServiceAccount token复现该错误，但不确定客户是否因该原因触发问题，也不清楚token过期/损坏的具体诱因。

失败Pod配置 #

apiVersion: v1
kind: Pod
metadata:
  name: abcd
  namespace: xyz
spec:
  containers:
  - command:
    - /bin/sh
    - -c
    - sleep infinity
    image: image-name
    imagePullPolicy: IfNotPresent
    name: test
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - SYS_ADMIN
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: true
      runAsNonRoot: false
      runAsUser: 0
    volumeMounts:
    - mountPath: /sample
      name: nfs
    - mountPath: /tmp
      name: tmp-volume
    - mountPath: /proxy-certs
      name: proxy-ca-cert
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-hh5l6
      readOnly: true
  imagePullSecrets:
  - name: f2d70900-3e63-4119--b8a9554b
  securityContext:
    runAsNonRoot: false
    runAsUser: 0
  serviceAccount: f2d70900-3e63-4119-9b8b-17644e82eebe
  volumes:
  - name: nfs
    persistentVolumeClaim:
      claimName: eb941bb0-4802-4f9d-b6bd-242709f52474
  - emptyDir: {}
    name: tmp-volume
  - configMap:
      defaultMode: 420
      name: f2d70900-3e63-4119-9b8b-17644e82eebe
    name: proxy-ca-cert
  - name: kube-api-access-hh5l6
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
      - configMap:
          items:
          - key: service-ca.crt
            path: service-ca.crt
          name: openshift-service-ca.crt

已尝试的复现步骤（未成功） #

• 将expirationSeconds缩短至10分钟，执行命令前添加10分钟睡眠
• 到期后Pod中serviceAccountToken已正确更新
• 到期后使用已创建的客户端可正常工作，client-go库已处理该场景

内容的提问来源于stack exchange，提问作者Shubham Yelikar