Kubernetes Dashboard访问异常与证书问题求助

Let's break down your problems step by step—they all trace back to worker04 using a self-generated CA for its kubelet certificates, which your cluster's master nodes don't trust. This breaks communication between the API server, kubelet, and Dashboard. Here's how to fix it:

1. Fix Worker04's Kubelet Certificate (Root Cause) #

Your API server can't validate worker04's kubelet because it's using a self-signed CA instead of the cluster's root CA. Let's regenerate the certificates properly:

On Master01 (Cluster CA Host): #

1. Locate your cluster's root CA files (usually in /etc/kubernetes/pki/):

   ls /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/ca.key
   

2. Generate a certificate signing request (CSR) for worker04. Create a config file worker04-kubelet-csr.conf with:

   [req]
   req_extensions = v3_req
   distinguished_name = req_distinguished_name
   [req_distinguished_name]
   [v3_req]
   basicConstraints = CA:FALSE
   keyUsage = nonRepudiation, digitalSignature, keyEncipherment
   extendedKeyUsage = serverAuth, clientAuth
   subjectAltName = @alt_names
   [alt_names]
   DNS.1 = worker04
   DNS.2 = worker04.your-cluster-domain.local # Replace with your actual domain
   IP.1 = <worker04-ip-address> # Replace with worker04's actual IP
   IP.2 = <worker04-internal-cluster-ip> # If using cluster IP, add it here
   

3. Generate the CSR and private key:

   openssl req -new -keyout worker04-kubelet.key -out worker04-kubelet.csr -config worker04-kubelet-csr.conf -nodes
   

4. Sign the CSR with the cluster's root CA:

   openssl x509 -req -in worker04-kubelet.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out worker04-kubelet.crt -days 3650 -extensions v3_req -extfile worker04-kubelet-csr.conf
   

On Worker04: #

1. Back up the existing kubelet certificates (in /var/lib/kubelet/pki/):

   sudo mv /var/lib/kubelet/pki/kubelet.crt /var/lib/kubelet/pki/kubelet.crt.bak
   sudo mv /var/lib/kubelet/pki/kubelet.key /var/lib/kubelet/pki/kubelet.key.bak
   sudo mv /var/lib/kubelet/pki/ca.crt /var/lib/kubelet/pki/ca.crt.bak
   

2. Copy the new worker04-kubelet.crt, worker04-kubelet.key, and the cluster's ca.crt from master01 to /var/lib/kubelet/pki/:

   # Use scp or your preferred method to transfer files
   scp master01:/path/to/worker04-kubelet.crt /var/lib/kubelet/pki/
   scp master01:/path/to/worker04-kubelet.key /var/lib/kubelet/pki/
   scp master01:/etc/kubernetes/pki/ca.crt /var/lib/kubelet/pki/
   

3. Set correct permissions:

   sudo chmod 600 /var/lib/kubelet/pki/kubelet.key
   sudo chown root:root /var/lib/kubelet/pki/*
   

4. Restart the kubelet service:

   sudo systemctl restart kubelet
   

Verify the Fix: #

Back on master01, check if worker04 is ready and the API server can communicate with it:

kubectl get nodes
kubectl logs -n kubernetes-dashboard kubernetes-dashboard-665f4c5ff-6qnzp # This should now work without x509 errors

2. Resolve Dashboard Access Timeout #

Once the certificate issue is fixed, address the dial tcp 10.36.0.1:8443: i/o timeout error:

• Restart Dashboard Pods: Refresh the Dashboard deployment to ensure it picks up the fixed cluster communication:

  kubectl rollout restart deployment kubernetes-dashboard -n kubernetes-dashboard
  

• Validate Load Balancer Configuration: Ensure your load balancer loadbalancer.local:6443 is correctly forwarding traffic to your master nodes' 6443 port. Test connectivity from the load balancer to master01:

  curl -k https://master01:6443 # Use -k to skip cert check temporarily for testing
  

• Check Dashboard Service: Confirm the Dashboard service's ClusterIP is reachable from master nodes:

  kubectl get svc kubernetes-dashboard -n kubernetes-dashboard
  curl -k https://<dashboard-cluster-ip>:8443 # Replace with the ClusterIP from the output
  

3. Fix Kubectl Proxy 403 Forbidden Error #

The 403 error when using kubectl proxy is likely due to host restrictions or RBAC misconfiguration:

• Allow All Hosts (Temporary Test): Restart the proxy with the --accept-hosts flag to bypass host checks:

  kubectl -v=9 proxy --port=8001 --address=192.168.1.24 --accept-hosts='.*'
  

• Verify RBAC Permissions: Ensure your kube-apiserver ServiceAccount has the correct cluster-admin binding:

  kubectl get clusterrolebindings | grep cluster-admin
  kubectl describe clusterrolebinding <your-cluster-admin-binding> -n kubernetes-dashboard # Replace with your binding name
  

• Check API Server Flags: Ensure your master nodes' kube-apiserver has flags like --allow-private-ips enabled if you're accessing from a private IP range like 192.168.x.x. You can check this in the kube-apiserver manifest (usually /etc/kubernetes/manifests/kube-apiserver.yaml).

内容的提问来源于stack exchange，提问作者tinashe.chipomho