Azure Kubernetes Service：配置带静态私有IP的内部负载均衡器

问题背景 #

使用以下Terraform代码创建的Azure Kubernetes Service（AKS）集群，默认生成的负载均衡器带有公网IP，希望配置为仅使用内部私有IP，避免服务通过公网IP暴露。尝试添加注解annotations: service.beta.kubernetes.io/azure-load-balancer-internal: "true"后，仍会分配公网IP，寻求解决方法。

原Terraform代码 #

# Required Provider
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0.2"
    }
  }
  required_version = ">= 1.1.0"
}

data "azurerm_client_config" "current" {}

provider "azurerm" {
  subscription_id = local.subscription_id
  tenant_id       = local.tenant_id
  client_id       = local.client_id
  client_secret   = local.client_secret

  features {}
}

resource "random_pet" "rg-name" {
  prefix = var.resource_group_name_prefix
}

resource "azurerm_resource_group" "rg" {
  name     = random_pet.rg-name.id
  location = var.resource_group_location
}

resource "azurerm_virtual_network" "test" {
  name                = var.virtual_network_name
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  address_space       = [var.virtual_network_address_prefix]

  subnet {
    name           = var.aks_subnet_name
    address_prefix = var.aks_subnet_address_prefix
  }

  tags = var.tags
}

data "azurerm_subnet" "kubesubnet" {
  name                 = var.aks_subnet_name
  virtual_network_name = azurerm_virtual_network.test.name
  resource_group_name  = azurerm_resource_group.rg.name
  depends_on           = [azurerm_virtual_network.test]
}

resource "azurerm_kubernetes_cluster" "k8s" {
  name       = var.aks_name
  location   = azurerm_resource_group.rg.location
  dns_prefix = var.aks_dns_prefix
  private_cluster_enabled = var.private_cluster
  resource_group_name = azurerm_resource_group.rg.name

  http_application_routing_enabled = false

  linux_profile {
    admin_username = var.vm_user_name

    ssh_key {
      key_data = file(var.public_ssh_key_path)
    }
  }

  default_node_pool {
    name            = "agentpool"
    node_count      = var.aks_agent_count
    vm_size         = var.aks_agent_vm_size
    os_disk_size_gb = var.aks_agent_os_disk_size
    vnet_subnet_id  = data.azurerm_subnet.kubesubnet.id
  }

  service_principal {
    client_id     = local.client_id
    client_secret = local.client_secret
  }

  network_profile {
    network_plugin     = "azure"
    dns_service_ip     = var.aks_dns_service_ip
    docker_bridge_cidr = var.aks_docker_bridge_cidr
    service_cidr       = var.aks_service_cidr
    load_balancer_sku  = "standard"
  }

  # Enabled the cluster configuration to the Azure kubernets with RBAC
  azure_active_directory_role_based_access_control { 
    managed                     = var.azure_active_directory_role_based_access_control_managed
    admin_group_object_ids      = var.active_directory_role_based_access_control_admin_group_object_ids
    azure_rbac_enabled          = var.azure_rbac_enabled
  }

  timeouts {
    create = "20m"
    delete = "20m"
  }  

  depends_on = [data.azurerm_subnet.kubesubnet,module.log_analytics_workspace]
  tags       = var.tags
}

解决方案 #

要实现AKS负载均衡器仅使用内部私有IP，需从集群层面修改默认负载均衡器配置，具体步骤如下：

1. 修改AKS集群的network_profile配置 #

在azurerm_kubernetes_cluster资源的network_profile块中，添加load_balancer_profile配置，明确禁用公网IP并指定内部负载均衡器的子网：

resource "azurerm_kubernetes_cluster" "k8s" {
  # 保留原有所有配置...

  network_profile {
    network_plugin     = "azure"
    dns_service_ip     = var.aks_dns_service_ip
    docker_bridge_cidr = var.aks_docker_bridge_cidr
    service_cidr       = var.aks_service_cidr
    load_balancer_sku  = "standard"

    # 添加负载均衡器专属配置
    load_balancer_profile {
      # 禁用AKS自动创建的公网出站IP
      outbound_type = "userDefinedRouting"
      managed_outbound_ip_count = 0

      # 指定内部负载均衡器使用的子网
      internal_load_balancer {
        subnet_id = data.azurerm_subnet.kubesubnet.id
      }
    }
  }

  # 保留原有所有配置...
}

2. 配置说明 #

• outbound_type = "userDefinedRouting"：告知AKS不自动创建公网IP用于出站流量，需自行通过NAT网关等方式配置出站路由
• managed_outbound_ip_count = 0：禁止AKS自动分配公网IP资源
• internal_load_balancer：指定负载均衡器绑定的子网，确保其仅在VNet内部分配私有IP

3. 后续Service配置（可选） #

后续创建LoadBalancer类型的Service时，添加注解service.beta.kubernetes.io/azure-load-balancer-internal: "true"，即可直接创建内部负载均衡器，不会生成公网IP。

注意事项 #

• 若集群需要访问公网资源，需额外配置NAT网关并关联到AKS所在子网，否则节点无法出站访问外部服务
• 必须使用标准SKU负载均衡器（代码中已设置load_balancer_sku = "standard"，满足要求）

内容的提问来源于stack exchange，提问作者One Developer