Terraform部署Prometheus时挂载ConfigMap卷出现超时错误

问题现象 #

本地使用Terraform部署Prometheus Kubernetes资源时，不配置卷挂载ConfigMap的情况下，资源能正常创建；但添加volumeMount和volume挂载ConfigMap后，部署超时并报错。

错误信息 #

│ Error: Deployment exceeded its progress deadline
│ 
│   with kubernetes_deployment.prometheus,
│   on prometheus.tf line 112, in resource "kubernetes_deployment" "prometheus":
│  112: resource "kubernetes_deployment" "prometheus" {
│ 
╵
2024-04-24T17:00:41.106+0500 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-04-24T17:00:41.108+0500 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/kubernetes/2.29.0/linux_amd64/terraform-provider-kubernetes_v2.29.0_x5 pid=112032
2024-04-24T17:00:41.108+0500 [DEBUG] provider: plugin exited

相关代码 #

ConfigMap定义 #

resource "kubernetes_config_map" "prometheus_config" {
  metadata {
    name = "prometheus-config"
  }
    data = {
        "prometheus.yml"= <<EOF

        */ 省略部分配置内容 */

        EOF
    }
}

Deployment定义 #

resource "kubernetes_deployment_v1" "prometheus" {
  metadata {
    name = "prometheus"
    labels = {
      app = "prometheus"
    }
  }

  depends_on = [kubernetes_config_map.prometheus_config]

  timeouts {
    create = "20m"
    update = "1h"
    delete = "20m"
  }

  spec {
    replicas = 1

    selector {
      match_labels = {
        app = "prometheus"
      }
    }

    template {
      metadata {
        labels = {
          app = "prometheus"
        }
      }

      spec {
        container {
          image = "prom/prometheus:latest"
          name  = "prometheus"
          port {
            container_port = 9090
          }
          
          # 经测试问题出在这个卷挂载配置

          volume_mount {
            name = "prometheus-config"
            mount_path = "/etc/prometheus"
            sub_path   = "prometheus.yml"
          }
        }
        
        volume {
          name = "prometheus-config"
          config_map {
            name = kubernetes_config_map.prometheus_config.metadata[0].name
          }
        }
      }
    }
  }
}

已排查情况 #

• 确认terraform apply时ConfigMap已成功创建，通过kubectl describe configmap prometheus-config验证内容与手动部署一致
• 注释卷挂载相关代码后，部署可在8秒内完成，结果如下：

Plan: 1 to add, 0 to change, 0 to destroy.

kubernetes_deployment_v1.prometheus: Creating...
kubernetes_deployment_v1.prometheus: Creation complete after 8s [id=default/prometheus]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

问题原因与解决方法 #

核心原因 #

当前卷挂载配置中，mount_path设为/etc/prometheus（Prometheus容器默认配置目录），并通过sub_path挂载单个文件。该目录下原本存在console_libraries、consoles等容器启动必需的文件/目录，这种挂载方式会导致容器无法访问这些必要资源，进而启动失败，最终引发Deployment超时。

解决方案1：直接挂载到默认配置文件路径 #

将mount_path指定为Prometheus默认配置文件的具体路径，而非整个目录，这样不会影响目录下的其他文件：

volume_mount {
  name = "prometheus-config"
  mount_path = "/etc/prometheus/prometheus.yml"
  sub_path   = "prometheus.yml"
}

解决方案2：挂载到自定义目录并指定启动参数 #

若需挂载多个配置文件，可将ConfigMap挂载到自定义目录，然后通过启动参数指定配置文件路径，同时保留容器原有必要资源的访问：

container {
  image = "prom/prometheus:latest"
  name  = "prometheus"
  port {
    container_port = 9090
  }
  # 添加启动参数指定自定义配置文件路径，同时保留默认的控制台资源路径
  args = [
    "--config.file=/etc/prometheus-custom/prometheus.yml",
    "--storage.tsdb.path=/prometheus",
    "--web.console.libraries=/etc/prometheus/console_libraries",
    "--web.console.templates=/etc/prometheus/consoles"
  ]
  volume_mount {
    name = "prometheus-config"
    mount_path = "/etc/prometheus-custom"
  }
}

额外检查项 #

验证ConfigMap中的prometheus.yml配置格式是否正确，可使用Prometheus官方工具本地检查：

promtool check config prometheus.yml

配置文件语法错误也会导致容器启动失败，引发Deployment超时。

内容的提问来源于stack exchange，提问作者stack_hopper