如何将Cloud Armor与Kong集成以增强部署安全性？

问题背景 #

需要将Cloud Armor与Kong集成以增强部署安全性，目标是创建Kong Ingress Controller，让多个服务共用附加Cloud Armor配置的同一负载均衡器，但尝试现有方案后未成功。

尝试过的配置 #

初始BackendConfig

➜  ~ devk describe BackendConfig
 Name:         cloudarmor-bitbucket
 Namespace:    default
 Labels:       <none>
 Annotations:  <none>
 API Version:  cloud.google.com/v1
 Kind:         BackendConfig
 Metadata:
   Creation Timestamp:  2023-07-26T19:13:05Z
   Generation:          1
   Resource Version:    33765445
   UID:                 some-uid
 Spec:
   Security Policy:
     Name:  bitbucket-pipelines-whitelist
 Events:    <none>

初始Kong Proxy Service片段

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{"default": "cloudarmor-bitbucket"}'
    cloud.google.com/neg: '{"ingress":true}'

完整Kong Proxy Service配置

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{"default": "bitbucket-pipelines-whitelist"}'
    cloud.google.com/neg: '{"ingress":true}'
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.beta.kubernetes.io/aws-load-balancer-backend-protocol":"tcp","service.beta.kubernetes.io/aws-load-balancer-type":"nlb"},"name":"kong-proxy","namespace":"kong-ext"},"spec":{"ports":[{"name":"proxy","port":80,"protocol":"TCP","targetPort":8000},{"name":"proxy-ssl","port":443,"protocol":"TCP","targetPort":8443}],"selector":{"app":"proxy-kong"},"type":"LoadBalancer"}}
  creationTimestamp: "2023-07-19T23:26:53Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  name: kong-proxy
  namespace: kong-ext
  resourceVersion: "11159902"
  uid: fd88a72b-0602-4398-9a0e-7ac4cd8366a0
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.118.0.217
  clusterIPs:
  - 10.118.0.217
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: proxy
    nodePort: 31413
    port: 80
    protocol: TCP
    targetPort: 8000
  - name: proxy-ssl
    nodePort: 31915
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    app: proxy-kong
  sessionAffinity: None
  type: LoadBalancer

对应BackendConfig完整配置

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cloud.google.com/v1","kind":"BackendConfig","metadata":{"annotations":{},"name":"bitbucket-pipelines-whitelist","namespace":"kong-ext"},"spec":{"securityPolicy":{"name":"bitbucket-pipelines-whitelist"}}}
  creationTimestamp: "2023-08-03T18:05:04Z"
  generation: 1
  name: bitbucket-pipelines-whitelist
  namespace: kong-ext
  resourceVersion: "11157046"
  uid: 2a882b8a-ad69-4375-8ae9-3523fa47f4df
spec:
  securityPolicy:
    name: bitbucket-pipelines-whitelist

修改为ClusterIP后的Kong Proxy Service

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{"default": "bitbucket-pipelines-whitelist"}'
    cloud.google.com/neg: '{"ingress":true}'
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.beta.kubernetes.io/aws-load-balancer-backend-protocol":"tcp","service.beta.kubernetes.io/aws-load-balancer-type":"nlb"},"name":"kong-proxy","namespace":"kong-ext"},"spec":{"ports":[{"name":"proxy","port":80,"protocol":"TCP","targetPort":8000},{"name":"proxy-ssl","port":443,"protocol":"TCP","targetPort":8443}],"selector":{"app":"proxy-kong"},"type":"LoadBalancer"}}
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  creationTimestamp: "2023-07-19T23:26:53Z"
  name: kong-proxy
  namespace: kong-ext
  resourceVersion: "14344187"
  uid: fd88a72b-0602-4398-9a0e-7ac4cd8366a0
spec:
  clusterIP: 10.118.0.217
  clusterIPs:
  - 10.118.0.217
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: proxy
    port: 80
    protocol: TCP
    targetPort: 8000
  - name: proxy-ssl
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    app: proxy-kong
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

正确集成步骤 #

1. 清理冲突配置 #

先移除Service中遗留的AWS相关注解，这些注解会干扰GCP负载均衡器的创建：

# 删除以下注解
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-type: nlb

2. 确保BackendConfig与Service同命名空间 #

GCP不支持跨命名空间绑定BackendConfig，必须保证两者处于同一命名空间（你的配置中已经修正为kong-ext，需维持此状态）。

3. 使用GCP Ingress暴露Kong（推荐方案） #

将Kong Proxy设为ClusterIP后，通过GCP Ingress创建公网负载均衡器并绑定BackendConfig：

3.1 确认Kong Proxy为ClusterIP

确保Kong Proxy Service的type为ClusterIP，且保留cloud.google.com/neg: '{"ingress":true}'注解，该注解会让GCP创建Network Endpoint Group（NEG）用于Ingress绑定。

3.2 创建GCP Ingress资源

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kong-ingress
  namespace: kong-ext
  annotations:
    kubernetes.io/ingress.class: "gce"
    cloud.google.com/backend-config: '{"default": "bitbucket-pipelines-whitelist"}'
spec:
  rules:
  - http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: kong-proxy
            port:
              number: 80

3.3 验证Cloud Armor绑定

创建Ingress后，GCP会自动创建负载均衡器并绑定安全策略，可通过以下命令验证：

gcloud compute backend-services describe <backend-service-name> --global

查看输出中的securityPolicy字段是否指向bitbucket-pipelines-whitelist。

4. 直接绑定到LoadBalancer Service（备选方案） #

若不需要Ingress，可直接在Kong Proxy的LoadBalancer Service上绑定BackendConfig：

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{"default": "bitbucket-pipelines-whitelist"}'
  name: kong-proxy
  namespace: kong-ext
spec:
  ports:
  - name: proxy
    port: 80
    protocol: TCP
    targetPort: 8000
  - name: proxy-ssl
    port: 443
    protocol: TCP
    targetPort: 8443
  selector:
    app: proxy-kong
  sessionAffinity: None
  type: LoadBalancer
  externalTrafficPolicy: Local

5. 验证配置生效 #

部署完成后，用不在白名单中的IP访问Kong公网IP，确认访问被拒绝，即可验证Cloud Armor策略生效。

内容的提问来源于stack exchange，提问作者Tiago Peres