Terraform部署AKS集群遇未授权及Kubernetes Provider不可达问题求助

问题描述 #

使用Terraform部署Azure AKS集群并配置Kubernetes资源，添加RBAC配置后出现以下授权错误：

module.kubernetes-config.kubernetes_namespace.test: Creating...
module.kubernetes-config.helm_release.nginx_ingress: Creating...

╷
│ Error: Unauthorized
│ 
│   with module.kubernetes-config.kubernetes_namespace.test,
│   on ../kubernetes-cluster/cluster.tf line 14, in resource "kubernetes_namespace" "test":
│   14: resource "kubernetes_namespace" "test" {
│ 
╵
╷
│ Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials
│ 
│   with module.kubernetes-config.helm_release.nginx_ingress,
│   on ../kubernetes-cluster/cluster.tf line 59, in resource "helm_release" "nginx_ingress":
│   59: resource "helm_release" "nginx_ingress" {
│ 
╵

根模块的Kubernetes Provider配置：

provider "kubernetes" {
  host                   = module.aks_cluster.host
  client_certificate     = base64decode(module.aks_cluster.client_certificate)
  client_key             = base64decode(module.aks_cluster.client_key)
  cluster_ca_certificate = base64decode(module.aks_cluster.cluster_ca_certificate)
}

provider "helm" {
  kubernetes {
    host                   = module.aks_cluster.host
    client_certificate     = base64decode(module.aks_cluster.client_certificate)
    client_key             = base64decode(module.aks_cluster.client_key)
    cluster_ca_certificate = base64decode(module.aks_cluster.cluster_ca_certificate)
  }
}

module "kubernetes-config" {
  depends_on   = [module.aks_cluster]
  source       = "../kubernetes-resources"
  kubeconfig   = module.aks_cluster.kube_config_raw
}

kubernetes-resources模块代码：

terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.0.3"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 2.1.0"
    }
  }
}

resource "kubernetes_namespace" "test" {
  metadata {
    name = "test"
  }
...
}

resource "kubernetes_deployment" "test" {
  metadata {
    name      = "test"
    namespace = kubernetes_namespace.test.metadata.0.name
  }
  
}

resource "helm_release" "nginx_ingress" {
  name = "nginx-ingress-controller"

  repository = "https://charts.bitnami.com/bitnami"
  chart      = "nginx-ingress-controller"

  set {
    name  = "service.type"
    value = "ClusterIP"
  }
}

resource "local_file" "kubeconfig" {
  content  = var.kubeconfig
  filename = "${path.root}/kubeconfig"
}

修复方案 #

1. 调整Kubernetes Provider的身份验证方式 #

如果AKS启用了Azure RBAC，客户端证书身份可能权限不足，建议改用Azure AD身份验证：

provider "kubernetes" {
  host                   = module.aks_cluster.host
  cluster_ca_certificate = base64decode(module.aks_cluster.cluster_ca_certificate)
  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "az"
    args = [
      "aks", "get-credentials",
      "--resource-group", module.aks_cluster.resource_group_name,
      "--name", module.aks_cluster.name,
      "--admin",
    ]
  }
}

provider "helm" {
  kubernetes {
    host                   = module.aks_cluster.host
    cluster_ca_certificate = base64decode(module.aks_cluster.cluster_ca_certificate)
    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "az"
      args = [
        "aks", "get-credentials",
        "--resource-group", module.aks_cluster.resource_group_name,
        "--name", module.aks_cluster.name,
        "--admin",
      ]
    }
  }
}

此方式通过az cli获取管理员凭证，确保拥有足够权限。

2. 确保模块继承正确的Provider配置 #

根模块配置的Provider可能未被kubernetes-config模块继承，可在kubernetes-resources模块内直接使用传入的kubeconfig配置Provider：

# 在kubernetes-resources模块中添加
provider "kubernetes" {
  config_content = var.kubeconfig
}

provider "helm" {
  kubernetes {
    config_content = var.kubeconfig
  }
}

这样模块内的资源会直接使用AKS生成的有效kubeconfig，避免凭证传递错误。

3. 验证凭证有效性 #

• 手动使用生成的kubeconfig测试集群连接：

  export KUBECONFIG=./kubeconfig
  kubectl get namespaces
  

  若命令失败，说明kubeconfig无效，需检查module.aks_cluster的输出变量是否正确传递。
• 确认client_certificate、client_key等变量解码后为合法的证书/密钥内容。

4. 检查RBAC权限配置 #

• 若启用Azure RBAC，确保执行Terraform的Azure AD身份拥有AKS集群管理员角色。
• 若使用本地RBAC，默认客户端证书对应的clusterAdmin用户拥有集群管理员权限，无需额外配置，若自定义了RBAC规则需确保该用户权限未被限制。

内容的提问来源于stack exchange，提问作者nocnoc