Akka Kubernetes API无法访问ServiceAccount令牌及Pod发现403问题求助

错误现象 #

我们使用Akka Kubernetes API连接AWS EKS集群中的其他Pod构建集群时，出现令牌读取权限错误：

[ERROR] 2022-12-01 12:05:53.635+0000 [cloud-poi-ha-akka.actor.default-dispatcher-11] a.d.k.KubernetesApiServiceDiscovery - Error reading api-token from /var/run/secrets/kubernetes.io/serviceaccount/token
java.nio.file.AccessDeniedException: /var/run/secrets/kubernetes.io/serviceaccount/token

检查Pod中挂载的令牌后，发现并非预期的sa-ne-ha服务账户令牌，而是默认令牌，导致Pod发现操作失败并返回403错误：

[INFO ] 2022-12-01 12:07:19.999+0000 [cloud-poi-ha-akka.actor.default-dispatcher-14] a.d.k.KubernetesApiServiceDiscovery - Querying for pods with label selector: [app=cloud-poi]. Namespace: [fdnbug-cloud-poi-ha-bugfix-def-85666]. Port: [None]
[WARN ] 2022-12-01 12:07:20.011+0000 [cloud-poi-ha-akka.actor.default-dispatcher-16] a.d.k.KubernetesApiServiceDiscovery - Forbidden to communicate with Kubernetes API server; check RBAC settings. Response: [{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:anonymous\" cannot list resource \"pods\" in API group \"\" in the namespace \"fdnbug-cloud-poi-ha-bugfix-def-85666\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
]
[WARN ] 2022-12-01 12:07:20.013+0000 [cloud-poi-ha-akka.actor.default-dispatcher-12] a.m.c.b.i.BootstrapCoordinator - Resolve attempt failed! Cause: akka.discovery.kubernetes.KubernetesApiServiceDiscovery$KubernetesApiException: Forbidden when communicating with the Kubernetes API. Check RBAC settings.

相关资源清单 #

我们配置的ServiceAccount、Role、RoleBinding及StatefulSet清单如下：

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: sa-ne-ha
  namespace: somenamespace
  labels:
    componentName: ne
    app: some-app
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ne-r-cluster-managers
  namespace: somenamespace
  labels:
    componentName: ne
    app: some-app
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/status
      - pods/log
      - statefulsets
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ne-rb-cluster-managers
  namespace: somenamespace
  labels:
    componentName: ne
    app: someapp
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ne-r-cluster-managers
subjects:
  - kind: ServiceAccount
    name: sa-ne-ha
    namespace: somenamespace
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ne-ha
  namespace: somenamespace
  labels:
    componentName: ne-ha
    app: someapp
    version: someversion
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "9252"
spec:
  serviceName: ne-headless
  replicas: 2
  selector:
    matchLabels:
      componentName: ne-ha
      app: someapp
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        componentName: ne-ha
        app: someapp
    spec:
      serviceAccountName: sa-ne-ha
      containers:
        - name: ne-ha
          image: ${DOCKER_IMAGE}
          imagePullPolicy: Always
          resources:
            requests:
              cpu: ${CONTAINERS_RESOURCES_CPU_REQUESTS}
              memory: ${CONTAINERS_RESOURCES_MEMORY_REQUESTS}
            limits:
              cpu: ${CONTAINERS_RESOURCES_CPU_LIMITS}
              memory: ${CONTAINERS_RESOURCES_MEMORY_LIMITS}
          command:
            - "/microservice/bin"
          env:
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 30
            failureThreshold: 3
            timeoutSeconds: 30
          livenessProbe:
            httpGet:
              path: /alive
              port: 8080
            initialDelaySeconds: 130
            periodSeconds: 30
            failureThreshold: 3
            timeoutSeconds: 5
---

提问 #

请问有没有人遇到过类似问题并成功解决？

内容的提问来源于stack exchange，提问作者Karan Khanna