Nomad Docker任务接入指定网络问题：无法解析Redpanda地址

运行Nomad作业时，Vector无法连接Redpanda，出现DNS解析失败错误：

Feb 27 08:57:01.931 ERROR rdkafka::client: librdkafka: Global error: Resolve (Local: Host resolution failure): redpanda-0:9092/bootstrap: Failed to resolve 'redpanda-0:9092': Name does not resolve (after 1ms in state CONNECT, 15 identical error(s) suppressed)

当前Nomad作业配置如下：

job "vector-redpanda" {
  datacenters = ["dc1"]
  # system job, runs on all nodes
  type = "system"
  update {
    min_healthy_time = "10s"
    healthy_deadline = "5m"
    progress_deadline = "10m"
    auto_revert = true
  }
  group "vector" {
    count = 1
    restart {
      attempts = 3
      interval = "10m"
      delay = "30s"
      mode = "fail"
    }
    network {
      port "api" {
        to = 8787
      }
      port "redpanda_network" {}
    }
    # docker socket volume
    volume "docker-sock-ro" {
      type = "host"
      source = "docker-sock-ro"
      read_only = true
    }
    ephemeral_disk {
      size    = 500
      sticky  = true
    }
    task "vector" {
      driver = "docker"
      config {
        image = "timberio/vector:0.14.X-alpine"
        ports = ["api", "redpanda_network"]
      }
      # docker socket volume mount
      volume_mount {
        volume = "docker-sock-ro"
        destination = "/var/run/docker.sock"
        read_only = true
      }
      # Vector won't start unless the sinks(backends) configured are healthy
      env {
        VECTOR_CONFIG = "local/vector.toml"
        VECTOR_REQUIRE_HEALTHY = "true"
      }
      # resource limits are a good idea because you don't want your log collection to consume all resources available
      resources {
        cpu    = 500 # 500 MHz
        memory = 256 # 256MB
      }
      # template with Vector's configuration
      template {
        destination = "local/vector.toml"
        change_mode   = "signal"
        change_signal = "SIGHUP"
        # overriding the delimiters to [[ ]] to avoid conflicts with Vector's native templating, which also uses {{ }}
        left_delimiter = "[["
        right_delimiter = "]]"
        data=<<EOH
                data_dir = "alloc/data/vector/"
              [api]
                enabled = true
                address = "0.0.0.0:8787"
                playground = true
              [sources.logs]
                type = "docker_logs"
              [transforms.records_for_redpanda]
                type = "lua"
                inputs = [ "logs" ]
                version = "2"
              [transforms.records_for_redpanda.hooks]
                process = """"
                  function (event, emit)
                    event.log.job_id = event.log.label.job_id
                    event.log.task_id = event.log.label.task_id
                    event.log.run_id = event.log.label.run_id
                    event.log.created_at = event.log.timestamp
                    if event.log.run_id ~= "" then
                      emit(event)
                    end
                  end""""
              [sinks.my_kafka_sink]
                type = "kafka"
                inputs = [ "records_for_redpanda" ]
                bootstrap_servers = "redpanda-0:9092"
                topic = "logs_aggregation"
                acknowledgements.enabled = true
                compression = "gzip"
                healthcheck.enabled = false
                encoding.only_fields = ["job_id", "task_id", "run_id", "message", "created_at"]
                        encoding.timestamp_format = "unix"
                        encoding.codec = "json"
                message_timeout_ms = 450000
        EOH
      }
      service {
        check {
          port     = "api"
          type     = "http"
          path     = "/health"
          interval = "30s"
          timeout  = "5s"
        }
      }
      kill_timeout = "30s"
    }
  }
}

手动创建Vector容器并加入Redpanda所在的Docker网络时，连接正常，但不知道如何在Nomad的Docker任务中实现这一操作。

要让Nomad部署的Vector容器加入Redpanda的Docker网络，可通过以下两种方式配置：

1. 加入自定义Docker网络 #

如果Redpanda运行在自定义Docker网络（例如名为redpanda-net），在Vector任务的config块中添加networks参数：

task "vector" {
  driver = "docker"
  config {
    image = "timberio/vector:0.14.X-alpine"
    ports = ["api", "redpanda_network"]
    # 加入Redpanda所在的自定义网络
    networks = ["redpanda-net"]
  }
  # 其余配置保持不变
}

注意：该自定义网络需在所有Nomad客户端节点上预先存在。

2. 使用Host网络模式 #

如果Redpanda采用host网络模式运行，将Vector任务的网络模式也设置为host：

task "vector" {
  driver = "docker"
  config {
    image = "timberio/vector:0.14.X-alpine"
    ports = ["api", "redpanda_network"]
    # 与Redpanda使用相同的host网络
    network_mode = "host"
  }
  # 其余配置保持不变
}

额外建议 #

• 如果Redpanda通过Nomad部署，建议将其注册为Consul服务，Vector的bootstrap_servers改用服务发现地址（如redpanda.service.consul:9092），这样即使Redpanda实例迁移，Vector也能自动解析到正确地址。
• 测试阶段可暂时将VECTOR_REQUIRE_HEALTHY设为false，避免因Redpanda未完全启动导致Vector启动失败，确认连接正常后再改回true。

内容的提问来源于stack exchange，提问作者Deeksha Sharma