配置Network Policy后Kubernetes CRD仍无法调用默认API服务

问题描述 #

我已安装External Secrets Operator（ESO）用于从AWS Parameter Store拉取密钥到Kubernetes集群，并为external-secrets命名空间配置了两个Network Policy：

1. 默认拒绝所有流量仅允许DNS的策略：

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-except-dns
  namespace: external-secrets
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress: []
  egress:
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: UDP
          port: 53
    - to:
        - namespaceSelector: {}
      ports:
        - protocol: TCP
          port: 53

2. 允许ESO pod间通信、访问Kubernetes基础设施及AWS的策略：

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-external-secrets
  namespace: external-secrets
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: external-secrets
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: external-secrets
          podSelector:
            matchLabels:
              app.kubernetes.io/part-of: external-secrets
  egress:
    # Internal ESO communication
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: external-secrets
          podSelector:
            matchLabels:
              app.kubernetes.io/part-of: external-secrets
    # Kubernetes infrastructure
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
    # AWS access
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
      ports:
        - protocol: TCP
          port: 443

但ESO Pod仍报错，无法连接Kubernetes API Server：

{"level":"error","ts":1779974730.0720608,"logger":"setup","msg":"unable to create managed secret client","error":"failed to determine if *v1.Secret is namespaced: failed to get restmapping: failed to get server groups: Get \"https://10.43.0.1:443/api\": dial tcp 10.43.0.1:443: i/o timeout","stacktrace":"github.com/external-secrets/external-secrets/cmd/controller.init.func2\n\t/home/runner/work/external-secrets/external-secrets/cmd/controller/root.go:205\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1019\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1148\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.10.1/command.go:1071\ngithub.com/external-secrets/external-secrets/cmd/controller.Execute\n\t/home/runner/work/external-secrets/external-secrets/cmd/controller/root.go:334\nmain.main\n\t/home/runner/work/external-secrets/external-secrets/main.go:27\nruntime.main\n\t/opt/hostedtoolcache/go/1.26.3/x64/src/runtime/proc.go:290"}

我尝试添加10.42.0.0/16 CIDR规则但无效，请问为何连接失败？

原因分析 #

1. Kubernetes API Server的ClusterIP不属于kube-system命名空间：你配置的allow-external-secrets策略中，针对Kubernetes基础设施的规则仅允许访问kube-system命名空间的Pod，但Kubernetes API Server的ClusterIP（这里是10.43.0.1）属于集群Service CIDR段（默认通常是10.43.0.0/16），并非kube-system命名空间内Pod的IP，因此该规则无法匹配API Server的访问请求。
2. 默认拒绝策略的优先级：deny-all-except-dns策略是默认拒绝所有流量，只有明确被其他策略允许的流量才能通过。由于你没有明确允许访问API Server ClusterIP的规则，ESO Pod无法发起对API Server的请求，导致超时。

解决方案 #

修改allow-external-secrets策略的egress部分，添加允许访问Kubernetes API Server ClusterIP段的规则：

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-external-secrets
  namespace: external-secrets
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: external-secrets
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: external-secrets
          podSelector:
            matchLabels:
              app.kubernetes.io/part-of: external-secrets
  egress:
    # Internal ESO communication
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: external-secrets
          podSelector:
            matchLabels:
              app.kubernetes.io/part-of: external-secrets
    # 允许访问Kubernetes API Server的ClusterIP段
    - to:
        - ipBlock:
            cidr: 10.43.0.0/16
      ports:
        - protocol: TCP
          port: 443
    # Kubernetes基础设施（kube-system内的Pod，比如CoreDNS等）
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
    # AWS access
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
      ports:
        - protocol: TCP
          port: 443

关键说明 #

• 替换10.43.0.0/16为你的集群实际Service CIDR（可通过kubectl cluster-info dump | grep service-cluster-ip-range查询）。
• 如果不确定Service CIDR，也可以直接添加针对10.43.0.1/32的规则，精准允许访问API Server的ClusterIP。

内容的提问来源于stack exchange，提问作者Maurice