迁移Istio至新节点组后istio-sidecar-injector服务找不到问题求助

Let’s dig into the possible causes of this frustrating issue—especially since you confirmed the service does exist. Here are the most likely culprits and how to check/fix them:

1. DNS Resolution Failures on the New Node Group #

Even though the istio-sidecar-injector service exists, the pods in your new node group might not be able to resolve its DNS name. This is a common gotcha when moving workloads to new nodes.

• Test DNS from a pod in the new node group:
  Spin up a temporary busybox pod targeted at your new node group and run a DNS lookup:

  kubectl run -it --rm busybox --image=busybox:1.28 --restart=Never --overrides='{"spec":{"nodeSelector":{"your-new-node-label":"value"}}}' -- nslookup istio-sidecar-injector.istio-system.svc.cluster.local
  

  If this fails to return the correct ClusterIP (10.100.85.254 in your case), your new nodes have DNS issues.
• Fix steps:
  • Verify CoreDNS pods are running and properly scheduled across your cluster: kubectl get pods -n kube-system -l k8s-app=kube-dns.
  • Check if new nodes have the correct DNS configuration (e.g., /etc/resolv.conf pointing to the cluster DNS service).
  • Ensure your CNI plugin is fully installed and functional on new nodes—broken CNI often causes DNS failures.

2. Misconfigured Mutating Webhook #

The sidecar injector’s webhook configuration might have a typo in the service name or namespace, even though the service itself exists.

• Inspect the webhook config:

  kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml
  

  Look for the clientConfig section under webhooks[0]. It should match this structure (double-check the name and namespace):

  clientConfig:
    service:
      name: istio-sidecar-injector
      namespace: istio-system
      path: /inject
      port: 443
  

  If any of these values are incorrect (e.g., namespace set to istio instead of istio-system), that’s the root cause.
• Fix steps:
  Edit the webhook config with kubectl edit mutatingwebhookconfiguration istio-sidecar-injector and correct the service details.

3. Network Policy Blocking Access #

If you have network policies in place, they might be preventing pods in the new node group from communicating with the istio-sidecar-injector service.

• Check for restrictive network policies:
  List all network policies in the istio-system namespace:

  kubectl get networkpolicies -n istio-system
  

  Look for policies that deny incoming traffic to the istio-sidecar-injector service, or policies in other namespaces that block outgoing traffic to istio-system.
• Fix steps:
  Temporarily delete any suspicious network policies to test if the issue resolves. If it does, update the policies to allow traffic between your new node group’s pods and the injector service.

4. Sidecar Injector Pods Aren’t Reachable from New Nodes #

Even if the service exists, the underlying injector pods might be running on old nodes that are unreachable from your new node group. Maybe you forgot to apply the same NodeSelector to the injector deployment?

• Check where injector pods are running:

  kubectl get pods -n istio-system -l app=sidecar-injector -o wide
  

  If they’re still on the old node group, and your new nodes have network isolation from the old group, that could cause connectivity issues.
• Fix steps:
  Update the istio-sidecar-injector deployment to include your new node group’s NodeSelector:

  kubectl edit deployment istio-sidecar-injector -n istio-system
  

  Add the NodeSelector under spec.template.spec, matching what you used for the ingressgateway.

5. TLS Certificate Issues (Less Likely, But Worth Checking) #

Sometimes TLS certificate problems can manifest as "service not found" errors because the connection is rejected before the service is properly resolved.

• Check injector pod certificates:
  Describe an injector pod to verify certificate volumes are mounted correctly:

  kubectl describe pod <injector-pod-name> -n istio-system
  

  Look for volumes like istio-certs or sidecar-injector-certs and confirm they’re mounted to the correct path (usually /etc/istio/certs).
• Fix steps:
  If certificates are missing or expired, reissue them using Istio’s istioctl tool:

  istioctl x create-credentials -n istio-system --type=https --secret-name=istio-sidecar-injector-certs --service-account=istio-sidecar-injector-service-account
  

Start with the DNS and webhook config checks first—those are the most common causes in scenarios like yours. Let me know if any of these lead you to the solution!

内容的提问来源于stack exchange，提问作者LobsterMan