GCP区域GKE集群+NEG+HTTP负载均衡的配置痛点与自动化诉求

感谢Gabriel Hodoroaga及其教程，我们在GCP中实现了以下流量链路配置：

互联网 > HTTP负载均衡器 > 网络端点组（NEG） > 单可用区GKE > NGINX入口控制器（ingress-nginx）

但将GKE从单可用区切换为区域集群后，通过gcloud命令手动重建配置存在两大问题：

• 仅初始部署有效，ingress-nginx Pod重启迁移至其他可用区后连接中断，需手动重新关联正确的NEG；
• 现有环境不使用gcloud命令，需通过GitHub Actions结合Terraform完成配置。

初始部署后配置正常运行，重启ingress-nginx Pod后，Pod迁移至其他可用区，后端仍绑定原NEG。

以下是基于Gabriel Hodoroaga教程的完整手动配置步骤（中文适配版）：

变量配置 #

至少需替换YOURDOMAIN变量为实际域名。

CLUSTER_NAME="lb-negs-nging-reg"
REGION="europe-west2"
YOURDOMAIN="请替换为你的域名"
echo $CLUSTER_NAME ; echo $REGION ; echo $YOURDOMAIN

创建集群 #

gcloud container clusters create $CLUSTER_NAME --region $REGION --machine-type "e2-medium" --enable-ip-alias --num-nodes=2 

添加ingress-nginx Helm仓库 #

helm repo update
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

安装ingress-nginx #

先创建values.regional.yaml配置文件：

cat << EOF > values.regional.yaml
controller:
  service:
    type: ClusterIP
    annotations:
      cloud.google.com/neg: '{"exposed_ports": {"80":{"name": "ingress-nginx-80-neg"}}}'
EOF

执行Helm安装命令：

helm install -f values.regional.yaml ingress-nginx ingress-nginx/ingress-nginx

部署测试用Web服务 #

准备测试服务配置文件dummy-app-lightweb.yaml：

cat << EOF > dummy-app-lightweb.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lightweb
spec:
  selector:
    matchLabels:
      app: dummy
  replicas: 3
  template:
    metadata:
      labels:
        app: dummy
    spec:
      containers:
      - name: lightweb
        image: alastairhm/alpine-lighttpd-php
        ports:
        - name: http
          containerPort: 80
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh", "-c", 'wget https://raw.githubusercontent.com/robinpecha/hello-world/main/php-header/index.php -P /var/www/']
---
apiVersion: v1
kind: Service
metadata:
  name: dummy-service
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: dummy
EOF

应用配置到集群：

kubectl apply -f dummy-app-lightweb.yaml 

验证测试服务状态：

kubectl get pods
kubectl port-forward lightweb-???????????-???? 8080:80
# 在浏览器访问 http://localhost:8080
# 按Ctrl+C停止端口转发

创建Ingress对象 #

准备Ingress配置文件dummy-ingress.yaml，需将域名DNS指向后续生成的负载均衡IP或修改本地hosts：

cat << EOF > dummy-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dummy-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: "$YOURDOMAIN"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dummy-service
            port:
              number: 80
EOF

应用Ingress配置：

kubectl apply -f dummy-ingress.yaml 

获取Ingress节点的网络标签和可用区 #

NETWORK_TAGS=$(gcloud compute instances list --filter="name=( $(kubectl get pod -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].spec.nodeName}') )" --format="value(tags.items[0])") ; echo $NETWORK_TAGS 
NODEZONE=$(gcloud compute instances list --filter="name=( $(kubectl get pod -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].spec.nodeName}') )" --format="value(zone)"); echo $NODEZONE 

配置防火墙规则 #

允许负载均衡源IP访问集群节点的80端口：

gcloud compute firewall-rules create $CLUSTER_NAME-lb-fw --allow tcp:80 --source-ranges 130.211.0.0/22,35.191.0.0/16 --target-tags $NETWORK_TAGS 

添加健康检查配置 #

创建针对ingress-nginx的HTTP健康检查：

gcloud compute health-checks create http app-service-80-health-check --request-path /healthz --port 80 --check-interval 60 --unhealthy-threshold 3 --healthy-threshold 1 --timeout 5 

创建后端服务 #

关联健康检查并创建全局后端服务：

gcloud compute backend-services create $CLUSTER_NAME-lb-backend --health-checks app-service-80-health-check --port-name http --global --enable-cdn --connection-draining-timeout 300 

关联NEG至后端服务 #

将ingress-nginx对应的NEG绑定到后端服务：

gcloud compute backend-services add-backend $CLUSTER_NAME-lb-backend --network-endpoint-group=ingress-nginx-80-neg --network-endpoint-group-zone=$NODEZONE --balancing-mode=RATE --capacity-scaler=1.0 --max-rate-per-endpoint=1.0 --global 

配置前端组件 #

创建URL映射、目标代理和转发规则，完成负载均衡前端配置：

gcloud compute url-maps create $CLUSTER_NAME-url-map --default-service $CLUSTER_NAME-lb-backend 
gcloud compute target-http-proxies create $CLUSTER_NAME-http-proxy --url-map $CLUSTER_NAME-url-map 
gcloud compute forwarding-rules create $CLUSTER_NAME-forwarding-rule --global --ports 80 --target-http-proxy $CLUSTER_NAME-http-proxy 

启用日志记录 #

开启后端服务的访问日志：

gcloud compute backend-services update $CLUSTER_NAME-lb-backend --enable-logging --global 

测试验证 #

获取负载均衡IP并验证访问：

IP_ADDRESS=$(gcloud compute forwarding-rules describe $CLUSTER_NAME-forwarding-rule --global --format="value(IPAddress)") ; echo $IP_ADDRESS
curl -s -I http://$IP_ADDRESS/ # 返回404（无域名匹配）
curl -s -I http://$YOURDOMAIN/ # 返回200（正常访问）

执行以下命令删除所有创建的资源：

gcloud -q compute forwarding-rules delete $CLUSTER_NAME-forwarding-rule --global 
gcloud -q compute target-http-proxies delete $CLUSTER_NAME-http-proxy 
gcloud -q compute url-maps delete $CLUSTER_NAME-url-map 
gcloud -q compute backend-services delete $CLUSTER_NAME-lb-backend --global
gcloud -q compute health-checks delete app-service-80-health-check
gcloud -q compute firewall-rules delete $CLUSTER_NAME-lb-fw 
kubectl delete -f dummy-ingress.yaml 
kubectl delete -f dummy-app-lightweb.yaml 
helm delete ingress-nginx 
gcloud -q container clusters delete $CLUSTER_NAME --zone=$ZONE
gcloud -q compute network-endpoint-groups delete ingress-nginx-80-neg --zone=$REGION-a
gcloud -q compute network-endpoint-groups delete ingress-nginx-80-neg --zone=$REGION-b
gcloud -q compute network-endpoint-groups delete ingress-nginx-80-neg --zone=$REGION-c
gcloud -q compute network-endpoint-groups list

内容的提问来源于stack exchange，提问作者rrob