Datadog Agent 7容器无法连接Redis服务器问题求助

问题背景 #

在Docker Compose中部署Datadog Agent 7作为服务后，容器日志出现两个错误：

1. redisdb检查报错连接超时，但未部署Redis服务
2. php_fpm检查提示必须设置status_url或ping_url，但未配置相关监控需求

原始Docker Compose配置 #

version: "3.9"

services:
  app:
    image: app
    container_name: app
    hostname: app
    build:
      context: .
      dockerfile: Dockerfile
    restart: unless-stopped
    ports:
      - 8080:80
    volumes:
      - shared_volume:/tmp/logs

  datadog:
    container_name: dd-agent
    image: gcr.io/datadoghq/agent:7
    restart: always
    ports:
      - 8125:8125/udp
      - 8126:8126
    environment:
      - DD_API_KEY=${DATADOG_API_KEY}
      - DD_SITE=${DD_SITE}
      - DD_DOGSTATSD_NON_LOCAL_TRAFFIC=${DD_DOGSTATSD_NON_LOCAL_TRAFFIC}
      - DD_LOGS_ENABLED="true"
      - DD_APM_ENABLED="true"
      - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL="true"
      - DD_CONTAINER_EXCLUDE_LOGS="name:dd-agent"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro

volumes:
  shared_volume:

错误日志 #

dd-agent  | 2022-10-11 10:13:53 UTC | CORE | ERROR | (pkg/collector/worker/check_logger.go:69 in Error) | check:php_fpm | Error running check: [{"message": "Detected 1 error while loading configuration model `InstanceConfig`:
__root__
  Field `status_url` or `ping_url` must be set", "traceback": "Traceback (most recent call last):
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 1091, in run
    initialization()
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 492, in load_configuration_models
    instance_config = self.load_configuration_model(package_path, 'InstanceConfig', raw_instance_config)
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 536, in load_configuration_model
    raise_from(ConfigurationError('\n'.join(message_lines)), None)
  File \"<string>\", line 3, in raise_from
datadog_checks.base.errors.ConfigurationError: Detected 1 error while loading configuration model `InstanceConfig`:
__root__
  Field `status_url` or `ping_url` must be set
"}]
dd-agent  | 2022-10-11 10:13:57 UTC | CORE | ERROR | (pkg/collector/worker/check_logger.go:69 in Error) | check:redisdb | Error running check: [{"message": "Timeout connecting to server", "traceback": "Traceback (most recent call last):
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/connection.py\", line 611, in connect
    sock = self.retry.call_with_retry(
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/retry.py\", line 51, in call_with_retry
    raise error
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/retry.py\", line 46, in call_with_retry
    return do()
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/connection.py\", line 612, in <lambda>
    lambda: self._connect(), lambda error: self.disconnect(error)
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/connection.py\", line 677, in _connect
    raise err
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/connection.py\", line 665, in _connect
    sock.connect(socket_address)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 1116, in run
    self.check(instance)
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/redisdb/redisdb.py\", line 556, in check
    self._check_db()
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/redisdb/redisdb.py\", line 205, in _check_db
    info = conn.info()
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/commands/core.py\", line 970, in info
    return self.execute_command(\"INFO\", **kwargs)
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/client.py\", line 1235, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/connection.py\", line 1387, in get_connection
    connection.connect()
  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/redis/connection.py\", line 615, in connect
    raise TimeoutError(\"Timeout connecting to server\")
redis.exceptions.TimeoutError: Timeout connecting to server
"}]

原因分析 #

• redisdb检查报错：Datadog Agent默认启用服务自动发现功能，通过挂载的Docker socket扫描容器时，可能误识别了端口或进程特征，自动触发redisdb检查，尝试连接不存在的Redis实例。
• php_fpm检查报错：同样是自动发现机制，Agent检测到app容器中的php-fpm相关进程，自动启用php_fpm检查，但未配置该检查必需的status_url或ping_url参数。

解决方案 #

方法一：直接禁用不需要的检查 #

在datadog服务的环境变量中添加DD_CHECKS_DISABLED，指定要禁用的检查项：

修改后的Docker Compose配置：

version: "3.9"

services:
  app:
    image: app
    container_name: app
    hostname: app
    build:
      context: .
      dockerfile: Dockerfile
    restart: unless-stopped
    ports:
      - 8080:80
    volumes:
      - shared_volume:/tmp/logs

  datadog:
    container_name: dd-agent
    image: gcr.io/datadoghq/agent:7
    restart: always
    ports:
      - 8125:8125/udp
      - 8126:8126
    environment:
      - DD_API_KEY=${DATADOG_API_KEY}
      - DD_SITE=${DD_SITE}
      - DD_DOGSTATSD_NON_LOCAL_TRAFFIC=${DD_DOGSTATSD_NON_LOCAL_TRAFFIC}
      - DD_LOGS_ENABLED="true"
      - DD_APM_ENABLED="true"
      - DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL="true"
      - DD_CONTAINER_EXCLUDE_LOGS="name:dd-agent"
      # 禁用不需要的检查
      - DD_CHECKS_DISABLED="redisdb,php_fpm"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro

volumes:
  shared_volume:

方法二：配置自动发现过滤规则（精细控制） #

如果需要保留其他自动发现功能，可通过环境变量过滤不需要的服务检查：

environment:
  # ... 其他原有变量
  # 排除指定服务的自动发现
  - DD_AUTODISCOVERY_EXCLUDE="name:redisdb,name:php_fpm"

验证解决效果 #

重启Docker Compose服务：

docker-compose down && docker-compose up -d

查看dd-agent日志，确认错误不再出现：

docker logs dd-agent

内容的提问来源于stack exchange，提问作者Aditya Mishra