Terraform+ECK管理的Elasticsearch集群版本升级失败排查

问题描述 #

生产环境原使用AWS手动管理的Elasticsearch集群，现通过Terraform部署Helm版ECK复刻集群，首次部署正常，但将ES版本从8.3.2升级至8.5.2时触发以下错误：

│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to kubernetes_manifest.elasticsearch_deploy, provider "provider["registry.terraform.io/hashicorp/kubernetes"]" produced an unexpected new
│ value: .object: wrong final value type: attribute "spec": attribute "nodeSets": tuple required.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

相关配置文件如下：

main.tf 核心配置 #

resource "kubernetes_manifest" "elasticsearch_deploy" {
  field_manager {
    force_conflicts = true
  }
  computed_fields = ["metadata.labels", "metadata.annotations", "spec.finalizers", "spec.nodeSets", "status"]
  manifest = yamldecode(templatefile("config/elasticsearch.yaml", {
    version                    = var.elastic_stack_version
    nodes                      = var.logging_elasticsearch_nodes_count
    cluster_name               = local.cluster_name
  }))
}

resource "kubernetes_manifest" "kibana_deploy" {
  field_manager {
    force_conflicts = true
  }
  depends_on      = [kubernetes_manifest.elasticsearch_deploy]
  computed_fields = ["metadata.labels", "metadata.annotations", "spec.finalizers", "spec.nodeSets", "status"]
  manifest = yamldecode(templatefile("config/kibana.yaml", {
    version                    = var.elastic_stack_version
    cluster_name               = local.cluster_name
    namespace                  = local.stack_namespace
  }))
}

Elasticsearch 模板（config/elasticsearch.yaml） #

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  annotations:
    eck.k8s.elastic.co/downward-node-labels: "topology.kubernetes.io/zone"
  name: ${cluster_name}
  namespace: ${namespace}
spec:
  version: ${version}
  volumeClaimDeletePolicy: DeleteOnScaledownAndClusterDeletion
  monitoring:
    metrics:
      elasticsearchRefs:
      - name: ${cluster_name}
    logs:
      elasticsearchRefs:
      - name: ${cluster_name}
  nodeSets:
    - name: logging-nodes
    count: ${nodes}
    config:
      node.store.allow_mmap: false]]

Kibana 模板（config/kibana.yaml） #

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: ${cluster_name}
  namespace: ${namespace}
spec:
  version: ${version}
  count: 1
  elasticsearchRef:
    name: ${cluster_name}
  monitoring:
    metrics:
      elasticsearchRefs:
        - name: ${cluster_name}
    logs:
      elasticsearchRefs:
        - name: ${cluster_name}
  podTemplate:
    metadata:
      labels:
        stack_name: ${stack_name}
        stack_repository: ${stack_repository}
    spec:
      serviceAccountName: ${service_account}
      containers:
        - name: kibana
          resources:
            limits:
              memory: 1Gi
              cpu: "1"

分析与解决 #

这不是ECK Operator的Bug，问题出在你的配置错误上，具体修正如下：

1. 修复Elasticsearch模板的YAML语法错误 #

模板中nodeSets的缩进完全错误，count和config没有嵌套在列表项- name: logging-nodes下，且末尾多了无效的]]。修正后的模板：

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  annotations:
    eck.k8s.elastic.co/downward-node-labels: "topology.kubernetes.io/zone"
  name: ${cluster_name}
  namespace: ${namespace}
spec:
  version: ${version}
  volumeClaimDeletePolicy: DeleteOnScaledownAndClusterDeletion
  monitoring:
    metrics:
      elasticsearchRefs:
      - name: ${cluster_name}
    logs:
      elasticsearchRefs:
      - name: ${cluster_name}
  nodeSets:
    - name: logging-nodes
      count: ${nodes}
      config:
        node.store.allow_mmap: false

该错误直接导致生成的nodeSets结构不符合CRD要求的数组（tuple）类型，触发Terraform的类型校验错误。

2. 修正Kibana的computed_fields配置 #

Kibana的CRD不存在spec.nodeSets字段，你却在kibana_deploy的computed_fields中添加了这个字段，会引发无效字段读取问题。修正后的配置：

resource "kubernetes_manifest" "kibana_deploy" {
  field_manager {
    force_conflicts = true
  }
  depends_on      = [kubernetes_manifest.elasticsearch_deploy]
  computed_fields = ["metadata.labels", "metadata.annotations", "spec.finalizers", "status"]
  manifest = yamldecode(templatefile("config/kibana.yaml", {
    version                    = var.elastic_stack_version
    cluster_name               = local.cluster_name
    namespace                  = local.stack_namespace
  }))
}

后续建议 #

• 修复上述错误后重新执行升级操作即可。
• force_conflicts = true可以保留，因为ECK Operator会自动管理部分字段（如状态、标签），该配置能避免Terraform与Operator的字段冲突。
• computed_fields需严格对应CRD的实际字段，不要添加不存在的字段，否则会引发类型或读取错误。

内容的提问来源于stack exchange，提问作者Roberto D'Arco