Spark-submit任务执行成功后仍不退出，需手动终止PID问题咨询

Hey there! I’ve run into this exact head-scratcher a few times, so let’s break down the most likely culprits and how to fix them:

• Check for unclosed custom thread pools/background threads
  Your code mentions a val pool = ...—if this is a custom ExecutorService or thread pool, it’s almost certainly the issue! Spark won’t terminate cleanly if non-daemon threads are still running in the background. Make sure to explicitly shut it down before stopping your SparkSession:

  // Assuming pool is an ExecutorService instance
  pool.shutdown()
  // Wait for threads to finish gracefully, or force stop if needed
  if (!pool.awaitTermination(10, java.util.concurrent.TimeUnit.SECONDS)) {
    pool.shutdownNow()
  }
  

• Use spark.stop() instead of just close()
  While close() is supposed to delegate to stop(), I’ve seen edge cases (especially with Hive support enabled) where stop() is more reliable for cleaning up all underlying resources. Wrap your session creation in a try-finally block to ensure it always gets called, even if an exception crashes your code:

  var spark: SparkSession = null
  try {
    spark = SparkSession
      .builder
      .enableHiveSupport()
      .config("spark.scheduler.mode", "FAIR")
      .appName("parjobs")
      .getOrCreate()
    
    // Your business logic here (Person case class, data processing, etc.)
  } finally {
    if (spark != null) {
      spark.stop() // This ensures Hive metastore connections, contexts are properly closed
    }
  }
  

• Look for unclosed external resources
  If your code interacts with databases, file streams, or other external systems, make sure those connections/streams are explicitly closed. A lingering database connection or open file can keep the Driver process alive even after Spark thinks it’s done processing.

• Check Spark Driver logs for blocked threads
  Dig into the Driver logs (use yarn logs -applicationId <your-app-id> for YARN mode, or console output for local mode) to look for messages like "waiting for" or thread dumps. This will tell you exactly which thread is hanging and why—super helpful for pinpointing hidden issues.

• Fair scheduler configuration edge cases
  Since you’re using FAIR scheduling, double-check that there are no pending tasks stuck in a scheduler pool. Sometimes tasks marked as "pending" can prevent the session from terminating—verify all jobs have completed successfully before stopping the session.

Give these steps a try, and odds are one of them will resolve the hanging session issue!

内容的提问来源于stack exchange，提问作者Jayadeep Jayaraman