Kubernetes Job绑定PVC提示删除但实际已绑定成功排障求助

Alright, let's work through this tricky issue where your PVC shows it's being deleted but actually exists and is bound, plus your Flink Job Pod isn't logging anything. Here's how to dig into each part step by step:

1. Fix the PVC's False "Being Deleted" State #

First, we need to resolve the state mismatch between what Kubernetes is reporting and reality:

• Verify the PVC's actual status: Run kubectl describe pvc flink-pv-claim-11 and check the Status field (it should say Bound) and the Events section. Look for any stale deletion events or errors that might be causing the false status.
• Check the linked PV: Find the PV bound to your PVC from the describe output, then run kubectl describe pv <your-pv-name>. Confirm the PV is also in Bound state and has no pending deletion events.
• Clear stuck finalizers: Kubernetes uses finalizers to ensure cleanup tasks complete, but they can get stuck. Check if there's an unexpected finalizer on the PVC:

  kubectl get pvc flink-pv-claim-11 -o yaml | grep -A5 finalizers
  

  If you see anything other than kubernetes.io/pvc-protection (or even if that one is stuck), you can safely remove it (after confirming no active cleanup is in progress) with:

  kubectl patch pvc flink-pv-claim-11 -p '{"metadata":{"finalizers":null}}'
  

• Refresh Kubernetes controller state: If the status still shows incorrectly, restart the kube-controller-manager (for single-node clusters) or trigger a rolling restart of the controller pods in managed clusters—this forces a state sync with etcd.

2. Diagnose the Silent Pod Issue #

A Pod with no logs is usually hiding a startup failure; let's uncover it:

• Check Pod events: Run kubectl describe pod <your-job-pod-name> and scan the Events section. Look for errors like:
  • PVC mount failures
  • Image pull issues
  • Permission denied errors
  • Insufficient node resources
• Check previous logs: Even if the current Pod is empty, previous crash instances might have logs. Run:

  kubectl logs <your-job-pod-name> --previous
  

• Test PVC accessibility manually: Spin up a temporary Pod to verify the PVC works correctly and has the right permissions:

  # Test basic mounting with busybox
  kubectl run -it --rm test-pvc --image=busybox --volume claimName=flink-pv-claim-11,mountPath=/test ls -la /test
  
  # Test with the same Flink user (9999) to check permissions
  kubectl run -it --rm test-pvc --image=flink:1.11.0-scala_2.11 --user=9999 --volume claimName=flink-pv-claim-11,mountPath=/test touch /test/testfile
  

  If the touch command fails, your PVC's directory permissions don't allow the Flink user to write, which will break the Job.
• Validate log configuration: Your Job uses a ConfigMap for log4j-console.properties. Double-check this file to ensure logs are being sent to stdout (the default for Kubernetes to capture logs). You can temporarily bypass the ConfigMap to test: modify your Job's container to add -Dlog4j.configuration=file:///opt/flink/conf/log4j-console.properties to the args, or use the default Flink log config.

3. Review Your Job YAML for Configuration Issues #

Looking at your provided YAML, a few things to check:

• Dual mounts of the same PVC: You're mounting job-artifacts-volume to both /opt/flink/usrlib and /opt/flink/data/job-artifacts. This is allowed, but ensure your PVC contains the required files in both paths (e.g., the Job JAR in /opt/flink/usrlib). If the JAR is missing, Flink will fail to start silently.
• Flink startup arguments: Your args include placeholders like <optional arguments> and <job arguments>. Make sure these are properly formatted (e.g., quoted strings, valid values) if you're using them. Try removing optional arguments temporarily to rule out syntax errors.
• Liveness probe timing: The liveness probe starts after 30 seconds and runs every 60 seconds. If your Job takes longer to start, the probe might kill the Pod before it can log anything. You could increase initialDelaySeconds to 60 or 90 as a test.

Final Steps to Resolve #

1. Fix the PVC state mismatch using the steps above.
2. Verify the PVC is accessible with the correct permissions for the Flink user.
3. Correct any startup issues in the Job (missing files, bad arguments, log config).
4. Delete the existing Job and reapply the fixed YAML to start fresh.

内容的提问来源于stack exchange，提问作者Dolphin