集成webhook-token与Kube-apiserver时遇'invalid bearer token'错误，如何解决？

问题背景 #

我参照K8s官方文档实现了Go语言的Token验证Webhook服务器，返回序列化后的TokenReview JSON字符串用于账号密码验证。服务器能正常返回200 OK响应，但K8s API Server始终提示"invalid bearer token"，验证无法通过。

kube-apiserver日志 #

I0530 12:17:07.582993       1 request.go:1188] Request Body: {"kind":"TokenReview","apiVersion":"authentication.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"token":"test:test1234","audiences":["https://kubernetes.default.svc.cluster.local"]},"status":{"user":{}}}
I0530 12:17:07.583057       1 round_trippers.go:463] POST https://webhook-token.nginx.svc.cluster.local:8080/?timeout=30s
I0530 12:17:07.583067       1 round_trippers.go:469] Request Headers:
I0530 12:17:07.583081       1 round_trippers.go:473]     Accept: application/json, */*
I0530 12:17:07.583091       1 round_trippers.go:473]     Content-Type: application/json
I0530 12:17:07.597571       1 round_trippers.go:574] Response Status: 200 OK in 14 milliseconds
I0530 12:17:07.597769       1 round_trippers.go:577] Response Headers:
I0530 12:17:07.597793       1 round_trippers.go:580]     Content-Type: Application/json
I0530 12:17:07.597840       1 round_trippers.go:580]     Content-Length: 199
I0530 12:17:07.597855       1 round_trippers.go:580]     Date: Tue, 30 May 2023 12:17:07 GMT
I0530 12:17:07.597941       1 request.go:1188] Response Body: {"apiVersion":"authentication.k8s.io/v1beta1","kind":"TokenReview","Status":{"authenticated":true,"user":{"username":"test","uid":"100"},"audiences":["https://kubernetes.default.svc.cluster.local"]}}
E0530 12:17:07.598332       1 authentication.go:70] "Unable to authenticate the request" err="invalid bearer token"

auth-config.yaml配置 #

apiVersion: v1
kind: Config
clusters:
  - name: authn
    cluster:
      server: https://webhook-token.nginx.svc.cluster.local:8080
      certificate-authority: /etc/pki/ca.pem
users:
  - name: kube-apiserver
    user:
      client-certificate: /etc/pki/client.pem # cert for the webhook plugin to use
      client-key: /etc/pki/client.key.pem          # key matching the cert
contexts:
- context:
    cluster: authn
    user: kube-apiserver
  name: authn
current-context: authn

kube-apiserver.yaml配置片段 #

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.128.0.3:6443
  creationTimestamp: null
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --v=8
    - --authentication-token-webhook-version=v1beta1
    - --authentication-token-webhook-config-file=/etc/auth-config.yaml
    - --advertise-address=10.128.0.3
    - --allow-privileged=true
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://kubernetes.default.svc.cluster.local
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    image: registry.k8s.io/kube-apiserver:v1.27.1
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 10.128.0.3
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: kube-apiserver
    readinessProbe:
      failureThreshold: 3
      httpGet:
        host: 10.128.0.3
        path: /readyz
        port: 6443
        scheme: HTTPS
      periodSeconds: 1
      timeoutSeconds: 15
    resources:
      requests:
        cpu: 250m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 10.128.0.3
        path: /livez
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/ca-certificates
      name: etc-ca-certificates
      readOnly: true
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /usr/local/share/ca-certificates
      name: usr-local-share-ca-certificates
      readOnly: true
    - mountPath: /usr/share/ca-certificates
      name: usr-share-ca-certificates
      readOnly: true
    - mountPath: /etc/auth-config.yaml
      name: webhook-token
      readOnly: true
    - mountPath: /etc/pki/ca.pem
      name: webhook-ca
      readOnly: true
    - mountPath: /etc/pki/client.pem
      name: client-cert
      readOnly: true
    - mountPath: /etc/pki/client.key.pem
      name: clientkey-cert
      readonly: true
  hostNetwork: true
  priority: 2000001000
  hostAliases:
  - ip: 10.98.16.119
    hostnames:
    - "webhook-token.nginx.svc.cluster.local"
  priorityClassName: system-node-critical
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  volumes:
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/ca-certificates
      type: DirectoryOrCreate
    name: etc-ca-certificates
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /usr/local/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-local-share-ca-certificates
  - hostPath:
      path: /usr/share/ca-certificates
      type: DirectoryOrCreate
    name: usr-share-ca-certificates
  - hostPath:
      path: /etc/auth-config.yaml
    name: webhook-token
  - hostPath:
      path: /etc/pki/ca.pem
    name: webhook-ca
  - hostPath:
      path: /etc/pki/client.pem
    name: client-cert
  - hostPath:
      path: /etc/pki/client.key.pem
    name: clientkey-cert
status: {}

问题原因 #

从API Server日志的响应体可以看到，返回的TokenReview JSON中**"Status"字段首字母大写**，而Kubernetes的API规范要求字段采用小驼峰命名（即status小写开头）。API Server的JSON反序列化逻辑严格区分大小写，无法识别大写的Status字段，因此会判定验证未通过，返回"invalid bearer token"错误。

解决方案 #

修改Go服务器的响应代码，将返回的TokenReview结构体中的Status字段改为小写开头的status，确保所有字段符合authentication.k8s.io/v1beta1版本的TokenReview结构定义。修改后的响应体应如下：

{"apiVersion":"authentication.k8s.io/v1beta1","kind":"TokenReview","status":{"authenticated":true,"user":{"username":"test","uid":"100"},"audiences":["https://kubernetes.default.svc.cluster.local"]}}

内容的提问来源于stack exchange，提问作者sw9719