应用PrometheusRule时出现网关超时问题排查求助

尝试修改集群中的PrometheusRule资源时遇到504网关超时错误，该规则在同版本的另一个集群中可正常应用。

待修改的PrometheusRule配置 #

# modif.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  annotations:
    meta.helm.sh/release-name: monitoring-platform
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2022-01-04T09:20:58Z"
  generation: 1
  labels:
    app: prometheus-operator
    app.kubernetes.io/managed-by: Helm
    prometheus: kube-op
    release: monitoring-platform
  name: kube-op-apps-rules
  namespace: monitoring
  resourceVersion: "948572193"
  uid: a461d478-9e61-4004-a129-9ed3f5efe8b0
spec:
  groups:
  - name: kubernetes-apps
    rules:
    - alert: KubePodCrashLooping
      annotations:
        message: Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container
          }}) is restarting {{ printf "%.2f" $value }} times / 20 minutes.
        runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodcrashlooping
      expr: rate(kube_pod_container_status_restarts_total{job="kube-state-metrics",
        pod!~"social-reco-prod.*"}[15m]) * 60 * 20 > 0
      for: 1h
      labels:
        severity: critical

Apply命令执行错误日志 #

09:14:23 bastien@work:/work/$ kubectl -v 6 apply -f modif.yaml
I0103 09:14:25.389081 1969468 loader.go:379] Config loaded from file:  /home/bastien/.kube/config
I0103 09:14:25.436584 1969468 round_trippers.go:445] GET https://1.2.3.4/openapi/v2?timeout=32s 200 OK in 46 milliseconds
I0103 09:14:25.655706 1969468 round_trippers.go:445] GET https://1.2.3.4/apis/external.metrics.k8s.io/v1beta1?timeout=32s 200 OK in 14 milliseconds
I0103 09:14:25.664871 1969468 cached_discovery.go:82] skipped caching discovery info, no resources found
I0103 09:14:25.696947 1969468 round_trippers.go:445] GET https://1.2.3.4/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules/kube-op-apps-rules 404 Not Found in 30 milliseconds
I0103 09:14:25.728817 1969468 round_trippers.go:445] GET https://1.2.3.4/api/v1/namespaces/monitoring 200 OK in 31 milliseconds
I0103 09:14:59.759927 1969468 round_trippers.go:445] POST https://1.2.3.4/apis/monitoring.coreos.com/v1/namespaces/monitoring/prometheusrules?fieldManager=kubectl-client-side-apply 504 Gateway Timeout in 34030 milliseconds

集群版本信息 #

# 故障集群
12:37:34 bastien@work:/work/$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:10:43Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14-gke.4300", GitCommit:"348bdc1040d273677ca07c0862de867332eeb3a1", GitTreeState:"clean", BuildDate:"2022-08-17T09:22:54Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}

# 正常集群
13:25:07 bastien@work:/work/$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:10:43Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.14-gke.4300", GitCommit:"348bdc1040d273677ca07c0862de867332eeb3a1", GitTreeState:"clean", BuildDate:"2022-08-17T09:22:54Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}

可能的问题点及排查方向 #

核心可能原因 #

• Prometheus Operator异常：PrometheusRule由Prometheus Operator管理，若Operator pod崩溃、内存不足或响应缓慢，会导致API Server处理请求超时。
• API Server资源瓶颈：故障集群的API Server可能CPU/内存耗尽、请求队列堆积，无法及时处理CRD资源的更新请求。
• CRD定义损坏：虽然K8s版本一致，但故障集群的prometheusrules.monitoring.coreos.com CRD可能存在结构损坏或版本不匹配。
• 网络链路问题：客户端到API Server、或API Server到Operator的网络延迟过高、链路不稳定。
• 资源锁冲突：目标资源可能被Helm或其他进程持有锁，或存在残留状态导致处理卡住。

具体排查步骤 #

1. 检查Prometheus Operator状态

   • 查看Operator pod运行状态：

     kubectl get pods -n monitoring -l app=prometheus-operator
     

   • 实时查看Operator日志，排查报错或卡顿：

     kubectl logs -n monitoring <prometheus-operator-pod-name> -f
     

2. 验证API Server健康状态

   • 查看集群组件状态：

     kubectl get componentstatuses
     

   • 通过GKE控制台或kubectl查看API Server的资源使用率，检查是否有超时、资源不足的日志。

3. 检查CRD完整性

   • 确认CRD存在且正常：

     kubectl get crd prometheusrules.monitoring.coreos.com
     

   • 导出CRD定义，与正常集群对比：

     kubectl get crd prometheusrules.monitoring.coreos.com -o yaml > crd-fault.yaml
     # 在正常集群执行相同命令后对比文件差异
     

4. 测试网络连通性

   • 客户端测试到API Server的延迟：

     ping 1.2.3.4
     curl -v https://1.2.3.4/healthz
     

   • 检查集群内部API Server到Operator pod的网络是否通畅。

5. 排除资源锁或冲突

   • 强制删除原资源后重新apply：

     kubectl delete prometheusrules kube-op-apps-rules -n monitoring --force --grace-period=0
     kubectl apply -f modif.yaml
     

   • 检查Helm release状态，确认无未完成操作：

     helm list -n monitoring
     helm status monitoring-platform -n monitoring
     

6. 简化测试验证

   • 创建极简PrometheusRule，测试是否能正常apply，排除规则本身问题：

     apiVersion: monitoring.coreos.com/v1
     kind: PrometheusRule
     metadata:
       name: test-rule
       namespace: monitoring
     spec:
       groups:
       - name: test-group
         rules:
         - alert: TestAlert
           expr: up == 1
           for: 1m
           labels:
             severity: warning
     

内容的提问来源于stack exchange，提问作者b.moyet