AWS EKS中Prometheus Adapter的APIService连接失败排查求助

问题描述 #

在AWS EKS Kubernetes集群中安装Prometheus adapter后，对应的APIService始终无法进入available状态。

执行kubectl edit apiservice v1beta1.custom.metrics.k8s.io后，获取到的状态错误信息如下：

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    meta.helm.sh/release-name: adapter-test
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2022-11-29T11:00:36Z"
  labels:
    app.kubernetes.io/component: metrics
    app.kubernetes.io/instance: adapter-test
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: prometheus-adapter
    app.kubernetes.io/part-of: prometheus-adapter
    app.kubernetes.io/version: v0.10.0
    helm.sh/chart: prometheus-adapter-3.4.2
  name: v1beta1.custom.metrics.k8s.io
  resourceVersion: "644865077"
  uid: 50e1d07b-5db8-49b0-92d3-af1ec581a096
spec:
  group: custom.metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: adapter-test-prometheus-adapter
    namespace: monitoring
    port: 443
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: "2022-11-29T11:00:36Z"
    message: 'failing or missing response from https://192.168.2.20:6443/apis/custom.metrics.k8s.io/v1beta1:
      Get "https://192.168.2.20:6443/apis/custom.metrics.k8s.io/v1beta1": Address
      is not allowed'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

集群版本信息：

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4", GitCommit:"872a965c6c6526caa949f0c6ac028ef7aff3fb78", GitTreeState:"clean", BuildDate:"2022-11-09T13:36:36Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.15-eks-fb459a0", GitCommit:"be82fa628e60d024275efaa239bfe53a9119c2d9", GitTreeState:"clean", BuildDate:"2022-10-24T20:33:23Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

本地通过curl可正常访问adapter服务：
执行命令：

curl -k --header "Authorization: Bearer <<MYTOKEN>>" "https://172.20.44.186:443/apis/custom.metrics.k8s.io/v1beta1"

返回响应：

{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "custom.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "namespaces/container_threads",
      "singularName": "",
      "namespaced": false,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "pods/container_threads",
      "singularName": "",
      "namespaced": true,
      "kind": "MetricValueList",
      "verbs": [
        "get"
      ]
    }
  ]
}

但adapter日志中未发现任何来自APIService的请求记录，需分析并解决该错误。

排查与解决步骤 #

1. 校验APIService的服务指向正确性 #

注意到APIService错误中访问的是192.168.2.20:6443，而本地curl访问的是172.20.44.186:443，两者地址不一致：

• 执行kubectl get svc adapter-test-prometheus-adapter -n monitoring，确认服务的ClusterIP是否为172.20.44.186
• 检查APIService的spec.service.port配置，确认是否误写为6443（Kubernetes API Server端口），需修正为adapter服务实际监听的443端口

2. 检查EKS集群的网络访问控制策略 #

AWS EKS中kube-apiserver访问内部服务可能受以下限制：

• VPC端点策略：若使用VPC端点访问EKS API，检查端点策略是否允许kube-apiserver访问monitoring命名空间下的服务
• 安全组配置：确认kube-apiserver所在安全组允许向adapter Pod的安全组发起443端口的访问
• 网络策略：检查monitoring命名空间是否存在阻止kube-apiserver访问的网络策略，可临时删除策略测试

3. 验证Prometheus Adapter的监听配置 #

检查adapter Pod的监听状态，确保其可被集群内部访问：

• 查看adapter Pod的启动命令，确认是否设置--secure-port=443，且未绑定到127.0.0.1
• 进入adapter Pod内部，执行netstat -tulpn，确认443端口在所有网卡上监听（显示0.0.0.0:443或:::443）

4. 排查kube-apiserver的访问限制 #

Kubernetes API Server的配置可能限制自定义APIService的访问：

• 通过CloudWatch查看EKS集群的API Server日志，搜索Address is not allowed关键词，获取更详细的错误上下文
• 确认Prometheus Adapter的服务账号已绑定system:auth-delegator角色，确保kube-apiserver能正确验证请求

5. 重新部署APIService与Adapter #

若以上检查均正常，尝试强制同步：

• 删除当前APIService：kubectl delete apiservice v1beta1.custom.metrics.k8s.io
• 重新安装或升级Helm Chart：helm upgrade --install adapter-test prometheus-community/prometheus-adapter -n monitoring
• 等待数分钟后，执行kubectl get apiservice v1beta1.custom.metrics.k8s.io查看状态是否恢复

内容的提问来源于stack exchange，提问作者Thomas0300