Datadog Agent无法发送Java追踪：K8s集群8126端口连接错误排查

问题背景 #

本地部署的K8s集群，所有组件已通过Datadog实现监控。当前Datadog APM仪表盘能展示请求及整体统计信息，但追踪（Trace）内容为空。按照Java自动注入Datadog APM库的相关步骤配置后，Pod出现如下连接错误日志。

Pod错误日志 #

[dd.trace 2023-01-26 07:40:33:818 +0000] [dd-trace-processor] WARN datadog.trace.agent.common.writer.ddagent.DDAgentApi - Error while sending 1 (size=2KB) traces. Total: 1, Received: 1, Sent: 0, Failed: 1. java.net.ConnectException: Failed to connect to /X.X.X.X:8126 (Will not log errors for 5 minutes)
[dd.trace 2023-01-26 07:41:23:838 +0000] [OkHttp http://X.X.X.X:8126/...] WARN com.datadog.profiling.uploader.ProfileUploader - Failed to upload profile to http://X.X.X.X:8126/profiling/v1/input java.net.ConnectException: Failed to connect to /X.X.X.X:8126 (Will not log errors for 5 minutes)

注：X.X.X.X为集群工作节点IP地址

相关配置信息 #

• Datadog Helm Chart配置文件已提供
• 测试用Java应用配置：

apiVersion: v1
kind: Pod
metadata:
  name: java
  annotations:
    admission.datadoghq.com/java-lib.version: "v1.5.0"
    admission.datadoghq.com/js-lib.version: "latest"
    admission.datadoghq.com/python-lib.version: "latest"
  labels:
    app: java
    tags.datadoghq.com/env: "dev"
    tags.datadoghq.com/service: "java"
    tags.datadoghq.com/version: "1.11.1"
    admission.datadoghq.com/enabled: "true"
spec:
  containers:
  - name: java
    image: nilushancosta/datadog-java-apm-demo
    ports:
    - name: java
      containerPort: 8080
    env:
      - name: DD_LOGS_INJECTION
        value: "true"
      - name: DD_ENV
        valueFrom:
          fieldRef:
            fieldPath: metadata.labels['tags.datadoghq.com/env']
      - name: DD_SERVICE
        valueFrom:
          fieldRef:
           fieldPath: metadata.labels['tags.datadoghq.com/service']
      - name: DD_VERSION
        valueFrom:
         fieldRef:
           fieldPath: metadata.labels['tags.datadoghq.com/version']
      - name: DD_SERVICE_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.labels['tags.datadoghq.com/service']
      - name: DD_PROFILING_ENABLED
        value: "true"
      - name: DD_TRACE_ANALYTICS_ENABLED
        value: "true"
      - name: DD_LOGS_INJECTION
        value: "true"
      - name: DD_TRACE_ENABLED
        value: "true"
      - name: DD_INSTRUMENTATION_TELEMETRY_ENABLED
        value: "true"
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: java
  name: java
  namespace: default
spec:
  selector:
    app: java
  ports:
  - name: java
    port: 80
    targetPort: 8080

内容的提问来源于stack exchange，提问作者Tutul Ckul