节点维护时，如何通过drain命令让Spark Driver Pod完成作业后再驱逐？

Great question—this is a common headache when you can’t modify Spark’s pod specs directly (since Kubernetes manages the YAML). Let’s walk through actionable, practical steps to protect your running jobs during node maintenance like kernel upgrades or hardware fixes:

1. First, Track When the Current Job Finishes #

You need clear visibility into whether the driver’s active job is done. Two reliable ways:

• Spark UI: Forward the driver pod’s 4040 port with kubectl port-forward <spark-driver-pod-name> 4040:4040, then check the "Jobs" tab to confirm all stages and tasks are marked as completed.
• Spark REST API: For scripting, query the driver’s API endpoint (e.g., curl http://<driver-pod-ip>:4040/api/v1/applications/<app-id>/jobs) to programmatically check job status. Use jq to parse the output and look for the final job’s status being "SUCCEEDED" or "FAILED" (either way, it’s finished processing).

2. Block New Jobs from Starting #

Before touching the node, make sure no new jobs get submitted to this driver. How you do this depends on your setup:

• If using the Spark Operator or Livy, pause job submissions to this specific driver instance.
• For standalone drivers, scale down any services that submit jobs to it, or temporarily block access to the driver’s submission port (e.g., 7077).

3. Cordon the Node First (Don’t Drain Yet) #

Run kubectl cordon <node-name> to prevent any new pods from being scheduled onto the node. This is critical because if the driver finishes its job and restarts (if it’s managed by a Deployment/StatefulSet), it won’t land back on the node you’re about to maintain.

4. Wait for the Active Job to Wrap Up #

Once the node is cordoned, monitor the driver until its job completes. You can automate this with a simple script loop:

while true; do
  # Fetch the latest job status using Spark API
  LATEST_JOB_STATUS=$(curl -s http://<driver-pod-ip>:4040/api/v1/applications/<app-id>/jobs | jq '.[-1].status')
  if [[ "$LATEST_JOB_STATUS" == "\"SUCCEEDED\"" || "$LATEST_JOB_STATUS" == "\"FAILED\"" ]]; then
    echo "Current job has finished. Ready to drain the node."
    break
  fi
  echo "Job still running—checking again in 30 seconds..."
  sleep 30
done

Pro tip: Adjust the sleep timer based on how long your typical jobs run. For long-running jobs, you might want to check every 5 minutes instead.

5. Drain the Node with a Generous Grace Period #

When you’re sure the job is done, run the drain command with an extended grace period to give the driver time to clean up resources properly:

kubectl drain <node-name> --grace-period=300 --ignore-daemonsets

The --ignore-daemonsets flag is safe here because Spark’s shuffle daemons (if present) can be rescheduled without impacting completed jobs.

6. Validate Post-Maintenance #

After draining, confirm:

• The driver pod has been rescheduled to a healthy node (if it’s managed by a controller).
• The completed job shows up as successful in your Spark history server (if you have one).

Quick Caveats to Keep in Mind #

• If your driver is a standalone pod (not managed by a Deployment/StatefulSet), you’ll need to manually restart it after maintenance if required—but only after the job is fully done.
• Always test this workflow in staging first! Production jobs don’t forgive mistakes.

内容的提问来源于stack exchange，提问作者shiv455