如何配置Otel-collector采集多副本业务Pod的监控指标

针对你的场景，有两种可行解决方案，其中Kubernetes服务发现是生产环境最优选择，能自动适配Pod扩缩容；静态Targets仅适合临时测试场景。

方案一：使用Kubernetes服务发现（推荐） #

通过Prometheus的Kubernetes服务发现机制，Otel-Collector会自动发现所有符合标签规则的业务Pod，无需手动维护Targets列表，完美适配Pod动态变化。

步骤1：更新Otel-Collector配置文件 #

修改otel-collector-conf ConfigMap中的Prometheus Receiver配置，替换原有static_configs为服务发现规则：

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-conf
  labels:
    app: opentelemetry
    component: otel-collector-conf
data:
  otel-collector-config: |
    receivers:
      prometheus:
        config:
          scrape_configs:
            - job_name: "mycoolbusinesspod"
              scrape_interval: 5s
              metrics_path: '/actuator/prometheus'
              # 启用Kubernetes Pod服务发现
              kubernetes_sd_configs:
                - role: pod
              # 筛选目标Pod：根据业务Pod的标签匹配（替换为你实际的Pod标签）
              relabel_configs:
                - source_labels: [__meta_kubernetes_pod_label_app]
                  action: keep
                  regex: mycoolbusinesspod
                # 仅采集Running状态的Pod
                - source_labels: [__meta_kubernetes_pod_phase]
                  action: keep
                  regex: Running
                # 拼接Pod IP与容器端口作为采集地址
                - source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_pod_container_port_number]
                  action: replace
                  target_label: __address__
                  regex: (.+);(.+)
                  replacement: ${1}:${2}
    
    processors:
      batch:
    
    exporters:
      prometheus:
        endpoint: "localhost:8889"
    
    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          processors: [batch]
          exporters: [prometheus]

步骤2：配置Kubernetes API访问权限 #

Otel-Collector需要权限访问K8s API获取Pod信息，创建对应的ServiceAccount、ClusterRole和绑定关系：

# 创建ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector
  namespace: default # 替换为你的实际命名空间

# 创建ClusterRole，允许读取Pod、节点、服务信息
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector-clusterrole
rules:
  - apiGroups: [""]
    resources: ["pods", "nodes", "services"]
    verbs: ["get", "list", "watch"]

# 绑定ClusterRole到ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-collector-clusterrolebinding
subjects:
  - kind: ServiceAccount
    name: otel-collector
    namespace: default # 与ServiceAccount命名空间一致
roleRef:
  kind: ClusterRole
  name: otel-collector-clusterrole
  apiGroup: rbac.authorization.k8s.io

步骤3：更新Otel-Collector Deployment关联ServiceAccount #

修改Deployment的Pod模板，指定刚才创建的ServiceAccount：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  labels:
    app: opentelemetry
    component: otel-collector
spec:
  selector:
    matchLabels:
      app: opentelemetry
      component: otel-collector
  minReadySeconds: 5
  progressDeadlineSeconds: 120
  replicas: 1
  template:
    metadata:
      labels:
        app: opentelemetry
        component: otel-collector
    spec:
      serviceAccountName: otel-collector # 新增该行关联ServiceAccount
      containers:
        - command:
            - "/otelcol"
            - "--config=/conf/otel-collector-config.yaml"
          image: otel/opentelemetry-collector:latest
          name: otel-collector
          resources:
            limits:
              cpu: 1
              memory: 2Gi
            requests:
              cpu: 200m
              memory: 400Mi
          ports:
            - containerPort: 55679 # ZPages默认端点
            - containerPort: 4317 # OpenTelemetry gRPC接收器默认端点
            - containerPort: 14250 # Jaeger gRPC接收器默认端点
            - containerPort: 14268 # Jaeger HTTP接收器默认端点
            - containerPort: 9411 # Zipkin接收器默认端点
            - containerPort: 8888  # 指标查询默认端点
          env:
            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: status.podIP
            - name: GOMEMLIMIT
              value: 1600MiB
          volumeMounts:
            - name: otel-collector-config-vol
              mountPath: /conf

方案二：静态Targets数组（仅临时测试） #

如果只是临时验证，可以手动将所有业务Pod的IP和端口添加到targets数组：

1. 获取业务Pod的IP列表：

kubectl get pods -l app=mycoolbusinesspod -o wide | awk '{print $6}' | grep -v IP

2. 修改ConfigMap中的static_configs：

static_configs:
  - targets: ["10.0.0.10:8080", "10.0.0.11:8080", "10.0.0.12:8080"] # 替换为实际Pod IP和端口

注意：该方案无法自动适配Pod扩缩容或重启，生产环境不推荐使用。

内容的提问来源于stack exchange，提问作者PatPanda