Prometheus无法发现Strimzi Cluster Operator的PodMonitor问题

环境架构 #

• Strimzi Operator部署在strimzi-system命名空间，监听命名空间A
• Kafka Broker、Exporter、Entity Operator部署在命名空间A
• kube-prometheus-stack（含Prometheus Operator）部署在monitoring命名空间
• 命名空间A内的Prometheus实例可正常抓取entity-operator-metrics和kafka-resources-metrics
• entity-operator-metrics与kafka-resources-metrics的PodMonitor部署在命名空间A，cluster-operator-metrics的PodMonitor部署在strimzi-system命名空间

已完成排查 #

• 确认Strimzi Cluster Operator Pod带有strimzi.io/kind: cluster-operator标签
• 验证monitoring命名空间的Prometheus服务账号拥有访问指标端点的ClusterRole权限
• 确认Strimzi Operator Pod的/metrics端点可正常返回指标
• 尝试将cluster-operator-metrics的PodMonitor部署到monitoring和命名空间A
• 取消Prometheus的PodMonitor发现限制，仍未找到目标

解决步骤 #

1. 对齐Prometheus与PodMonitor的标签选择器 #

Prometheus仅会抓取匹配自身podMonitorSelector标签的PodMonitor资源：

• 查看monitoring命名空间下Prometheus的标签选择器配置：

  kubectl get prometheus -n monitoring kube-prometheus-stack-prometheus -o jsonpath='{.spec.podMonitorSelector}'
  

• 给strimzi-system下的cluster-operator-metrics PodMonitor添加匹配标签（示例：若Prometheus用release: kube-prometheus-stack作为选择条件）：

  kubectl label podmonitor cluster-operator-metrics -n strimzi-system release=kube-prometheus-stack
  

2. 确保Prometheus允许发现strimzi-system命名空间的PodMonitor #

检查Prometheus的podMonitorNamespaceSelector配置，确保覆盖strimzi-system：

• 若配置了命名空间选择器，给strimzi-system添加对应标签：

  kubectl label namespace strimzi-system monitoring=enabled
  

• 或者将podMonitorNamespaceSelector设为空（允许发现所有命名空间的PodMonitor），修改Prometheus CR：

  spec:
    podMonitorNamespaceSelector: {}
  

3. 验证PodMonitor的配置正确性 #

检查cluster-operator-metrics PodMonitor的spec是否与Cluster Operator Pod匹配：

• 确认selector.matchLabels正确匹配Cluster Operator Pod的标签（默认strimzi.io/kind: cluster-operator）
• 确认podMetricsEndpoints的端口和路径与Pod的metrics端口一致（默认端口名metrics，路径/metrics）
  示例正确配置片段：

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: cluster-operator-metrics
  namespace: strimzi-system
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      strimzi.io/kind: cluster-operator
  podMetricsEndpoints:
  - port: metrics
    path: /metrics

4. 检查NetworkPolicy是否阻断访问 #

若strimzi-system存在NetworkPolicy，需添加规则允许monitoring命名空间的Prometheus Pod访问Cluster Operator的metrics端口（默认9404）：

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus-access
  namespace: strimzi-system
spec:
  podSelector:
    matchLabels:
      strimzi.io/kind: cluster-operator
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: monitoring
    ports:
    - protocol: TCP
      port: 9404

5. 确认服务账号权限覆盖strimzi-system #

确保Prometheus服务账号绑定的ClusterRole包含对strimzi-system命名空间下pods、endpoints资源的访问权限：

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-kube-prometheus-stack
rules:
- apiGroups: [""]
  resources: ["pods", "endpoints"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["monitoring.coreos.com"]
  resources: ["podmonitors"]
  verbs: ["get", "list", "watch"]

内容的提问来源于stack exchange，提问作者Kostya Zgara