Spark 2.3在K8s集群执行spark-submit时遇到两类异常问题

Hey there, I’ve dealt with similar headaches running Spark on K8s before, so let’s break down your two issues and get you sorted out.

Issue 1: Persistent Warnings That Don’t Interrupt Execution #

First up, those warnings that keep popping up but don’t kill your app—they’re not harmless long-term, even if they don’t break things right away. Here’s what’s likely going on and how to fix it:

• Possible Causes:

  • Overly verbose logging levels in Spark (default settings might be spitting out info-level messages that look like warnings)
  • Minor mismatches between your Spark version and Kubernetes cluster version (e.g., deprecated API calls that Spark still uses)
  • Low-level resource pressure on K8s nodes (like disk space warnings or minor CPU throttling that Spark picks up)
  • Misconfigured sidecar containers or volume mounts in your Executor Pod specs

• Fixes to Try:

  1. Tune Spark Logging: Edit your log4j.properties or log4j2.xml file to raise the log level for the noisy components. For example, if you’re seeing warnings from org.apache.spark.deploy.k8s, set it to ERROR instead of WARN:

     log4j.logger.org.apache.spark.deploy.k8s=ERROR
     

  2. Validate Version Compatibility: Double-check that your Spark version is officially compatible with your K8s version (Spark 3.x generally works with K8s 1.21+, but always confirm the docs). If there’s a minor mismatch, upgrading either component might quiet the warnings.
  3. Check Node Health: Run kubectl describe nodes to look for any persistent resource warnings (like low disk, memory pressure) on the nodes running your Executors. Clean up unused volumes or resize nodes if needed.
  4. Audit Pod Specs: If you’re customizing the Executor Pod template, make sure volume mounts, environment variables, and sidecars are correctly configured—typos or missing permissions can trigger warning spam.

Issue 2: Intermittent Executor Pod Startup Failures (Kills Spark-Submit) #

This is the more critical one since it’s causing your spark-submit to fail occasionally. Intermittent failures usually point to race conditions or resource-related flakiness in K8s. Here’s how to diagnose and fix it:

• Possible Causes:

  • Insufficient cluster resources (nodes don’t have enough CPU/memory to schedule Executors when your job starts)
  • Flaky image pulls (network issues with your container registry, or missing image tags)
  • RBAC permission gaps (your Spark ServiceAccount doesn’t have enough permissions to create Executor Pods consistently)
  • Network policy restrictions (sometimes network rules block communication between Driver and Executor during startup)
  • Too-short startup timeouts (Spark gives up waiting for Executors to start before they can fully initialize)

• Fixes to Try:

  1. Check Resource Availability: Use kubectl top nodes to monitor resource usage when you submit jobs. If nodes are hitting resource limits, consider adding more nodes, adjusting your Executor resource requests/limits, or setting up cluster autoscaling.
  2. Stabilize Image Pulls:
     • Ensure your Spark image is hosted in a registry that’s reliably accessible from your K8s cluster (no intermittent network blips)
     • Use fixed image tags (not latest) to avoid unexpected image changes
     • Add image pull secrets to your Spark ServiceAccount if the registry is private:

       kubectl create secret docker-registry reg-secret --docker-server=<registry-url> --docker-username=<user> --docker-password=<pass>
       

       Then reference it in your spark-submit: --conf spark.kubernetes.container.image.pullSecrets=reg-secret
  3. Verify RBAC Permissions: Make sure your Spark ServiceAccount has the right roles to create, delete, and modify Pods. A basic ClusterRole for Spark should include permissions like pods/create, pods/delete, pods/get, pods/list, pods/watch. You can bind it with:

     kubectl create clusterrolebinding spark-role-binding --clusterrole=spark-cluster-role --serviceaccount=<namespace>:spark-sa
     

  4. Adjust Startup Timeouts: Increase the Executor startup timeout to give pods more time to initialize, especially if your cluster is under load. Add this to your spark-submit command:

     --conf spark.kubernetes.executor.startTimeout=300s
     

     (Default is 120s—bumping it to 5 minutes can help with intermittent delays)
  5. Check Network Policies: If you have network policies enabled in your namespace, ensure they allow inbound/outbound traffic between the Spark Driver Pod and Executor Pods. At minimum, allow traffic on the ports Spark uses for communication (usually 7077, 4040, and random high ports for Executors).
  6. Enable Retries: Spark has built-in retries for Executor failures. Enable them with:

     --conf spark.executor.instances=3 --conf spark.kubernetes.executor.retry.count=2
     

     This lets Spark retry starting failed Executors a couple of times before giving up.

Hopefully these steps help you squash both issues—let me know if you need to dive deeper into any specific part!

内容的提问来源于stack exchange，提问作者shiv455