Windows环境下Apache Spark中\tmp\hive的作用及路径修改问题

Hey there! Let me walk you through this since I’ve wrestled with Windows-Spark setup more times than I can count.

This directory acts as a Windows-compatible stand-in for the Unix/Linux default /tmp/hive path, and it serves three critical roles:

• Storing temporary query intermediate results: When Spark runs SQL queries (especially those involving shuffles or large datasets), it generates temporary files to hold intermediate data. This directory is the default storage spot for these transient files.
• Managing Hive metadata locks: Spark SQL relies on Hive’s metadata system under the hood. To prevent concurrent processes from corrupting metadata (like table schema changes), Hive uses lock files—winutils creates and manages these locks in \tmp\hive to ensure operation consistency.
• Emulating Unix-like file system behavior: The Hadoop ecosystem was built for Unix/Linux, so many components expect a standard temporary directory structure. Winutils uses \tmp\hive to mimic that environment on Windows, ensuring Spark and Hive components work seamlessly together without compatibility gaps.

Absolutely! You’re not stuck with the default path, but you need to configure it properly to avoid headaches:

• Update Hive/Spark configuration: You have two reliable ways to set a custom path:
  1. In your Spark code: When initializing the SparkSession, add a config for hive.exec.scratchdir:

     val spark = SparkSession.builder()
       .appName("MySparkApp")
       .config("hive.exec.scratchdir", "D:\\my-custom-tmp\\hive")
       .enableHiveSupport()
       .getOrCreate()
     

  2. Via hive-site.xml: For a global, cross-job configuration, add this property to your Spark’s conf/hive-site.xml file:

     <property>
       <name>hive.exec.scratchdir</name>
       <value>D:/my-custom-tmp/hive</value>
       <description>Custom scratch directory for Hive/Spark SQL operations</description>
     </property>
     

• Watch out for permissions: No matter which path you choose, make sure the user running Spark has full read, write, and execute permissions on that directory. Winutils enforces Unix-style permission checks, so missing permissions will throw "Permission denied" errors faster than you can troubleshoot.
• Avoid problematic paths: Steer clear of directories with Chinese characters, spaces, or overly long paths—these often cause encoding or path parsing bugs that are a nightmare to debug.

内容的提问来源于stack exchange，提问作者Rinky