使用Terraform实现AKS Baselines：AGW无法连接AKS内部LB

问题描述 #

我正在用Terraform实现AKS基线架构，遇到一个问题：无法将应用网关（AGW）连接到AKS创建的内部负载均衡器（Internal Load Balancer）。

我的AKS集群部署了Solr实例，对应的Service配置了azure-load-balancer-internal注解，AKS与内部LB位于同一子网，应用网关则在同一VNet的独立子网中。

Kubernetes Service配置 (kubernetes.tf) #

resource "kubernetes_service" "solr-service" {
  metadata {
    name        = local.solr.name
    annotations = {
      "service.beta.kubernetes.io/azure-load-balancer-internal" : "true"
      "service.beta.kubernetes.io/azure-load-balancer-internal-subnet" : "aks-subnet"
    }
  }

  spec {
    external_traffic_policy = "Local"
    selector = {
      app = kubernetes_deployment.solr.metadata.0.labels.app
    }

    port {
      name        = "http"
      port        = 80
      target_port = 8983
    }

    type             = "LoadBalancer"
    load_balancer_ip = "192.168.1.200"
  }
}

上述配置在MC_*资源组中创建了前端IP为192.168.1.200的内部LB，其健康检查指标显示100%正常，说明LB本身工作正常。

应用网关配置 (application-gateway.tf) #

resource "azurerm_application_gateway" "agw" {
  name                = local.naming.agw_name
  resource_group_name = azurerm_resource_group.this.name
  location            = azurerm_resource_group.this.location

  sku {
    name     = "Standard_Medium"
    tier     = "Standard"
    capacity = 1
  }

  gateway_ip_configuration {
    name      = "Gateway-IP-Config"
    subnet_id = azurerm_subnet.agw_snet.id
  }

  frontend_port {
    name = "http-port"
    port = 80
  }

  frontend_ip_configuration {
    name                 = "public-ip"
    public_ip_address_id = azurerm_public_ip.agw_ip.id
  }

  backend_address_pool {
    name         = "lb"
    ip_addresses = ["192.168.1.200"]
  }

  backend_http_settings {
    name                  = "settings"
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 60
  }

  http_listener {
    name                           = "http-listener"
    frontend_ip_configuration_name = "public-ip"
    frontend_port_name             = "http-port"
    protocol                       = "Http"
  }

  request_routing_rule {
    name                       = local.request_routing_rule_name
    rule_type                  = "Basic"
    http_listener_name         = "http-listener"
    backend_address_pool_name  = "lb"
    backend_http_settings_name = "settings"
  }
}

问题现象 #

预期应用网关能连接到内部LB并转发请求，但后端池始终显示不健康，网关无法访问该内部IP。

已尝试的排查动作：

• 将应用网关用作Ingress控制器，可正常工作，但希望通过内部LB实现
• 给后端节点池添加健康检查，无效果
• 将LB改为公网类型，使用公网IP添加到应用网关后一切正常，说明问题出在内部LB的访问上，但同一VNet的子网默认应允许通信，未设置任何限制

解决方案 #

1. 检查并修改AKS子网的NSG规则 #

AKS会自动为子网创建NSG，可能默认限制了来自应用网关子网的流量。需添加入站规则允许应用网关子网访问内部LB的80端口：

resource "azurerm_network_security_rule" "allow_agw_to_lb" {
  name                        = "Allow-AGW-to-LB"
  priority                    = 100
  direction                   = "Inbound"
  access                      = "Allow"
  protocol                    = "Tcp"
  source_port_range           = "*"
  destination_port_range      = "80"
  source_address_prefix       = azurerm_subnet.agw_snet.address_prefix
  destination_address_prefix  = "192.168.1.200/32"
  network_security_group_name = azurerm_network_security_group.aks_nsg.name
  resource_group_name         = azurerm_resource_group.mc_rg.name # 替换为你的MC_开头的资源组名称
}

2. 配置匹配Solr的健康检查探针 #

应用网关默认健康检查路径为/，但Solr的健康检查路径应为/solr/admin/info/system或对应版本的健康端点，路径不匹配会导致后端标记为不健康：

resource "azurerm_application_gateway" "agw" {
  # 保留原有配置...

  # 新增健康检查探针
  probe {
    name                = "solr-health-probe"
    protocol            = "Http"
    path                = "/solr/admin/info/system"
    interval            = 30
    timeout             = 10
    unhealthy_threshold = 3
    healthy_threshold   = 2
  }

  backend_http_settings {
    name                  = "settings"
    cookie_based_affinity = "Disabled"
    port                  = 80
    protocol              = "Http"
    request_timeout       = 60
    probe_name            = "solr-health-probe" # 关联探针
  }

  # 保留原有配置...
}

3. 使用内部LB的FQDN作为后端目标 #

参考AKS基线架构示例，使用内部LB的FQDN替代IP，避免IP相关的解析或配置问题：

# 首先通过数据源获取AKS内部LB的信息
data "azurerm_lb" "aks_internal_lb" {
  name                = "kubernetes-internal" # AKS创建的内部LB默认名称
  resource_group_name = azurerm_resource_group.mc_rg.name # MC_开头的资源组
}

# 修改应用网关后端池配置
resource "azurerm_application_gateway" "agw" {
  # 保留原有配置...

  backend_address_pool {
    name         = "lb"
    fqdns        = [data.azurerm_lb.aks_internal_lb.frontend_ip_configuration.0.private_ip_address_fqdn]
  }

  # 保留原有配置...
}

4. 确认子网路由表配置 #

检查应用网关子网和AKS子网的路由表，确保没有自定义路由规则阻止了子网间的流量通信。如果存在指向NVA或其他地址的默认路由，需调整规则允许子网间的正常访问。

内容的提问来源于stack exchange，提问作者vscoding