基于多外部IP的Kube-Proxy配置问题：服务绑定特定IP异常

Hey there, let's dive into fixing this multi-NIC externalIPs problem you're facing. I've dealt with similar routing conflicts between custom iproute2 rules and Kubernetes components before, so here's what's likely going wrong and how to resolve it:

Common Root Causes & Fixes #

1. Kube-Proxy's IPTables/IPVS Rules Are Overriding Your Custom Routes #

By default, kube-proxy injects a ton of IPTables (or IPVS) rules to handle Service traffic routing—these can easily clash with your pre-configured iproute2 tables and rules, especially for externalIPs.

Fix Steps:

• Identify kube-proxy's mode: First, check if it's using IPTables or IPVS:

  kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode
  

• Exclude your external IP ranges from kube-proxy's rules:
  Edit the kube-proxy ConfigMap to exclude the IPs assigned to eth1 and eth2, so kube-proxy doesn't mess with their routing:

  # kubectl edit configmap kube-proxy -n kube-system
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: kube-proxy
    namespace: kube-system
  data:
    config.conf: |
      apiVersion: kubeproxy.config.k8s.io/v1alpha1
      kind: KubeProxyConfiguration
      mode: "iptables" # or "ipvs" if that's your mode
      iptables:
        excludeCIDRs:
        - "<eth1-public-ip>/32"
        - "<eth2-public-ip>/32"
      # If using IPVS, add this instead:
      # ipvs:
      #   excludeCIDRs:
      #   - "<eth1-public-ip>/32"
      #   - "<eth2-public-ip>/32"
  

• Restart kube-proxy to apply changes:

  kubectl rollout restart daemonset kube-proxy -n kube-system
  

2. Kubelet Isn't Recognizing Multiple External IPs #

Kubelet defaults to using a single node IP (often the first detected one, like eth0), which can break externalIPs binding to eth1/eth2 because Kubernetes doesn't know those IPs belong to the node.

Fix Steps:

• Update kubelet's startup parameters:
  Edit your kubelet service config (usually at /etc/systemd/system/kubelet.service.d/10-kubeadm.conf) to add both external IPs:

  # Add this to the KUBELET_EXTRA_ARGS line
  --node-external-ips=<eth1-public-ip>,<eth2-public-ip>
  

• Restart kubelet:

  systemctl daemon-reload && systemctl restart kubelet
  

3. Service Traffic Is Being Forwarded to Other Nodes (Breaking Return Path) #

If your Service uses the default externalTrafficPolicy: Cluster, traffic to your externalIP might get forwarded to a Pod on another node—this breaks your iproute2 return path rules, since the return traffic won't go through the original eth1/eth2 interface.

Fix Steps:

• Set externalTrafficPolicy: Local on your Services:
  This ensures traffic only goes to Pods running on the same node, keeping the return path aligned with your custom routing rules:

  apiVersion: v1
  kind: Service
  metadata:
    name: my-target-service
  spec:
    type: ClusterIP
    externalIPs:
    - "<eth1-public-ip>" # Bind to eth1
    ports:
    - port: 80
      targetPort: 8080
    externalTrafficPolicy: Local
  

  Repeat this for your other Service bound to eth2, swapping the externalIP.

Verification Steps #

After making these changes, verify everything works:

• Check iproute2 rules are still intact:

  ip rule show
  ip route show table <your-custom-table-id>
  

• Test traffic to each externalIP:

  curl <eth1-public-ip>:<service-port>
  curl <eth2-public-ip>:<service-port>
  

• Use tcpdump on eth1/eth2 to confirm traffic is entering and exiting the correct interface:

  tcpdump -i eth1 host <eth1-public-ip>
  

内容的提问来源于stack exchange，提问作者Thubo