Windows11下用Spark-XML读XML文件遇FileNotFoundError求助

在Windows 11系统的Python环境中，尝试使用Spark-XML读取园艺问答数据集的XML文件时，触发FileNotFoundError: [WinError 2] 系统找不到指定文件错误。

运行代码 #

from pyspark.sql import SparkSession
def main():
    gardening_raw_path = "D:\dbfs\dbdemos\product\llm\gardening\raw"
    print(f"loading raw xml dataset under {gardening_raw_path}")

    # Create a SparkSession
    spark = SparkSession.builder \
        .appName("module1") \
        .getOrCreate()

    # Read XML file into a DataFrame
    xml_df = spark.read \
        .format("xml") \
        .option("rowTag", "row") \
        .load(f"{gardening_raw_path}/Posts.xml")

    # Show the DataFrame schema and contents
    xml_df.printSchema()
    xml_df.show()

    # Stop the SparkSession
    spark.stop()

if __name__ == '__main__':
    main()

报错堆栈信息 #

Traceback (most recent call last):
  File "C:\Users\hp\Documents\module1.py", line 38, in <module>
    main()
  File "C:\Users\hp\Documents\module1.py", line 20, in main
    .getOrCreate()
     ^^^^^^^^^^^^^
  File "D:\Ali\python\Lib\site-packages\pyspark\sql\session.py", line 477, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Ali\python\Lib\site-packages\pyspark\context.py", line 512, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "D:\Ali\python\Lib\site-packages\pyspark\context.py", line 198, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "D:\Ali\python\Lib\site-packages\pyspark\context.py", line 432, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
                                       ^^^^^^^^^^^^^^^^^^^^
  File "D:\Ali\python\Lib\site-packages\pyspark\java_gateway.py", line 99, in launch_gateway
    proc = Popen(command, **popen_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Ali\python\Lib\subprocess.py", line 1024, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "D:\Ali\python\Lib\subprocess.py", line 1509, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified

从报错堆栈可以看出，错误发生在Spark启动Java网关阶段，并非直接找不到XML文件，以下是针对性修复步骤：

• 配置Java运行环境
  Spark依赖Java，需先安装JDK（推荐1.8或11版本），然后配置系统环境变量：

  1. 添加JAVA_HOME变量，值为JDK安装路径（例如C:\Program Files\Java\jdk1.8.0_381）
  2. 将%JAVA_HOME%\bin加入系统PATH变量
  3. 重启命令行/IDE，执行java -version确认环境生效

• 修复Windows路径转义问题
  Python字符串中的反斜杠\会被解析为转义字符，导致实际路径错误，修改为以下两种方式之一：

  # 方式1：使用原始字符串前缀r
  gardening_raw_path = r"D:\dbfs\dbdemos\product\llm\gardening\raw"
  # 方式2：用正斜杠替代反斜杠
  gardening_raw_path = "D:/dbfs/dbdemos/product/llm/gardening/raw"
  

• 完善Spark环境配置

  1. 确认已安装PySpark：执行pip install pyspark
  2. 若PySpark无法自动识别Spark安装包，添加系统环境变量SPARK_HOME，值为Spark解压路径（例如D:\spark-3.5.0-bin-hadoop3）
  3. 将%SPARK_HOME%\bin加入系统PATH变量

• 添加Spark-XML依赖
  创建SparkSession时需指定Spark-XML的依赖包，版本需与Spark版本兼容（例如Spark 3.5.x对应0.17.0）：

  spark = SparkSession.builder \
      .appName("module1") \
      .config("spark.jars.packages", "com.databricks:spark-xml_2.12:0.17.0")
      .getOrCreate()
  

内容的提问来源于stack exchange，提问作者Ali Raza