Terraform Cloud二次运行时无法访问AKS集群问题排查

问题描述 #

首次运行Terraform脚本部署AKS集群和Helm Chart正常，但后续运行时出现集群不可达错误，报错信息如下：

Error: Kubernetes cluster unreachable: invalid configuration: no configuration has 
been provided, try setting KUBERNETES_MASTER environment variable
with helm_release.nginx
on kubernetes.tf line 38, in resource "helm_release" "nginx":

对应的Terraform Cloud脚本：

resource "azurerm_kubernetes_cluster" "Example_Test" {
  name                = "Cluster_Example_Test"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  dns_prefix          = "Example_Test"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_D2_v2"
  }

  identity {
    type = "SystemAssigned"
  }

  tags = {
    Environment = "Development"
  }
}

data "azurerm_kubernetes_cluster" "Example_Test" {
  name                = azurerm_kubernetes_cluster.Example_Test.name
  resource_group_name = azurerm_kubernetes_cluster.Example_Test.resource_group_name
  depends_on          = [azurerm_kubernetes_cluster.Example_Test]
}

provider "helm" {

  kubernetes {
    host                   = 
data.azurerm_kubernetes_cluster.Example_Test.kube_config.0.host
    client_certificate     = base64decode(data.azurerm_kubernetes_cluster.Example_Test.kube_config.0.client_certificate)
    client_key             = base64decode(data.azurerm_kubernetes_cluster.Example_Test.kube_config.0.client_key)
    cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.Example_Test.kube_config.0.cluster_ca_certificate)
  }
}

resource "helm_release" "nginx" {
  name       = "nginx"
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  provider = helm
  set {
    name  = "controller.replicaCount"
    value = "1"
  }

  depends_on = [azurerm_kubernetes_cluster.Example_Test]
}

问题原因 #

1. Provider初始化时机问题：Terraform的depends_on仅作用于资源，无法直接控制Provider的初始化顺序。后续运行时，Helm Provider可能在AKS集群的kube_config未就绪前就完成初始化，导致配置缺失。
2. 冗余Data资源：额外使用data.azurerm_kubernetes_cluster获取集群信息，增加了依赖层级，反而容易引发加载时序问题。

修复方案 #

直接使用AKS集群资源的输出配置Helm Provider，并为Provider添加显式依赖，确保集群就绪后再初始化Helm。修复后的脚本如下：

resource "azurerm_kubernetes_cluster" "Example_Test" {
  name                = "Cluster_Example_Test"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  dns_prefix          = "Example_Test"

  default_node_pool {
    name       = "default"
    node_count = 1
    vm_size    = "Standard_D2_v2"
  }

  identity {
    type = "SystemAssigned"
  }

  tags = {
    Environment = "Development"
  }
}

provider "helm" {
  kubernetes {
    host                   = azurerm_kubernetes_cluster.Example_Test.kube_config.0.host
    client_certificate     = base64decode(azurerm_kubernetes_cluster.Example_Test.kube_config.0.client_certificate)
    client_key             = base64decode(azurerm_kubernetes_cluster.Example_Test.kube_config.0.client_key)
    cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.Example_Test.kube_config.0.cluster_ca_certificate)
  }

  # 显式声明依赖，确保AKS集群完全就绪后再初始化Helm Provider
  depends_on = [azurerm_kubernetes_cluster.Example_Test]
}

resource "helm_release" "nginx" {
  name       = "nginx"
  repository = "https://kubernetes.github.io/ingress-nginx"
  chart      = "ingress-nginx"
  provider   = helm

  set {
    name  = "controller.replicaCount"
    value = "1"
  }
}

修复说明 #

• 移除冗余的data.azurerm_kubernetes_cluster资源，直接引用AKS集群资源的kube_config输出，简化依赖逻辑。
• 在Helm Provider中添加depends_on，强制Provider等待AKS集群创建完成并输出有效配置后再初始化，彻底解决后续运行时的配置缺失问题。
• 移除helm_release中的depends_on，通过Provider的依赖已能保证部署顺序，简化脚本结构。

内容的提问来源于stack exchange，提问作者Ben Gannaway