PySpark本地保存DataFrame到Parquet报错，是否必须安装Hadoop？

问题描述 #

我刚安装了PySpark，由于PySpark官方文档未推荐安装Hadoop，因此未安装该组件。现咨询：在本地机器保存Parquet文件时，是否所有人都必须安装Hadoop？

我的代码 #

from datetime import datetime, date
import pandas as pd
from pyspark.sql import Row, SparkSession

spark = SparkSession.builder.appName("Example").getOrCreate()
df = spark.createDataFrame([
      Row(a=1, b=2., c='string1', d=date(2000, 1, 1), e=datetime(2000, 1, 1, 12, 0)),
      Row(a=2, b=3., c='string2', d=date(2000, 2, 1), e=datetime(2000, 1, 2, 12, 0)),
      Row(a=4, b=5., c='string3', d=date(2000, 3, 1), e=datetime(2000, 1, 3, 12, 0))
])
df.write.parquet('bar.parquet') # error 

运行环境 #

• Windows 11
• Python 3.11.8
• PySpark 3.5.0
• VS Code Notebook

报错信息 #

---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
Cell In[1], line 12
      5 spark = SparkSession.builder.appName("Example").getOrCreate()
      6 df = spark.createDataFrame([
      7     Row(a=1, b=2., c='string1', d=date(2000, 1, 1), e=datetime(2000, 1, 1, 12, 0)),
      8     Row(a=2, b=3., c='string2', d=date(2000, 2, 1), e=datetime(2000, 1, 2, 12, 0)),
      9     Row(a=4, b=5., c='string3', d=date(2000, 3, 1), e=datetime(2000, 1, 3, 12, 0))
     10 ])
---> 12 df.write.parquet('bar.parquet')

File ~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python311\\site-packages\\pyspark\\sql\\readwriter.py:1721, in DataFrameWriter.parquet(self, path, mode, partitionBy, compression)
   1719     self.partitionBy(partitionBy)
   1720 self._set_opts(compression=compression)
-> 1721 self._jwrite.parquet(path)

File ~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python311\\site-packages\\py4j\\java_gateway.py:1322, in JavaMember.__call__(self, *args)
   1316 command = proto.CALL_COMMAND_NAME +\\
   1317     self.command_header +\\
   1318     args_command +\\
   1319     proto.END_COMMAND_PART
   1321 answer = self.gateway_client.send_command(command)
-> 1322 return_value = get_return_value(
   1323     answer, self.gateway_client, self.target_id, self.name)
   1325 for temp_arg in temp_args:
   1326     if hasattr(temp_arg, "_detach"):

File ~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python311\\site-packages\\pyspark\\errors\\exceptions\\captured.py:179, in capture_sql_exception.<locals>.deco(*a, **kw)
    177 def deco(*a: Any, **kw: Any) -> Any:
    178     try:
--> 179         return f(*a, **kw)
    180     except Py4JJavaError as e:
    181         converted = convert_exception(e.java_exception)

File ~\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python311\\site-packages\\py4j\\protocol.py:326, in get_return_value(answer, gateway_client, target_id, name)
    324 value = OUTPUT_CONVERTER[type](answer[2:], gateway_client)
    325 if answer[1] == REFERENCE_TYPE:
--> 326     raise Py4JJavaError(
    327         "An error occurred while calling {0}{1}{2}.\\n".
    328         format(target_id, ".", name), value)
    329 else:
    330     raise Py4JError(
    331         "An error occurred while calling {0}{1}{2}. Trace:\\n{3}\\n".
    332         format(target_id, ".", name, value))

Py4JJavaError: An error occurred while calling o43.parquet.
: java.lang.RuntimeException: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems

解答 #

不用安装完整的Hadoop套件。你遇到的报错是Windows环境下PySpark依赖Hadoop的Winutils工具导致的，而非必须安装整套Hadoop。

解决方法 #

1. 配置Winutils（推荐）

• 下载与你的Spark版本匹配的Winutils二进制文件（比如Spark 3.5.0对应Hadoop 3.x版本的winutils）
• 新建一个文件夹（例如C:\hadoop），在其中创建bin子目录，将winutils.exe放入bin目录
• 系统环境变量中新增HADOOP_HOME，值设为C:\hadoop
• 将%HADOOP_HOME%\bin添加到系统PATH环境变量中
• 重启VS Code后重新运行代码

2. 配置Spark使用本地文件系统（备选）

在创建SparkSession时添加配置项，强制使用本地文件系统实现：

spark = SparkSession.builder \
    .appName("Example") \
    .config("spark.hadoop.fs.file.impl", "org.apache.hadoop.fs.LocalFileSystem") \
    .getOrCreate()

注意：此方法在部分Spark版本中可能不稳定，优先推荐第一种方案。

内容的提问来源于stack exchange，提问作者bouachalazhar