使用PyArrow引擎将Dask DataFrame写入HDFS Parquet时遇属性错误

Hey there, that error pops up because of a breaking API change in PyArrow—let’s break down how to fix it quickly:

Why This Happens #

Starting with PyArrow 0.17.0, the old pyarrow.hdfs module was deprecated and replaced with the pyarrow.fs module (specifically pyarrow.fs.HadoopFileSystem). If your Dask version is older, it might still be trying to call the deprecated pyarrow.hdfs API, which no longer exists in newer PyArrow releases.

Solutions to Try #

1. Manually Pass a Hadoop Filesystem Instance to Dask #

Instead of letting Dask auto-detect the HDFS connection, create a HadoopFileSystem object yourself and pass it to the filesystem parameter of to_parquet:

import pyarrow.fs

# Configure these based on your HDFS cluster settings
hdfs_fs = pyarrow.fs.HadoopFileSystem(
    host="your-namenode-host",  # e.g., "namenode.example.com"
    port=8020,  # Default HDFS RPC port, adjust if yours is different
    user="your-username"  # Optional, if you need to run as a specific user
)

# Write the Dask DataFrame with the explicit filesystem
dask_df.to_parquet(parquet_path, engine="pyarrow", filesystem=hdfs_fs)

Make sure your Hadoop config files (core-site.xml, hdfs-site.xml) are accessible to your Python process—you can set the HADOOP_CONF_DIR environment variable to the directory containing these files if needed.

2. Upgrade Dask to a Newer Version #

Newer versions of Dask (2021.06.0 and later) are compatible with PyArrow's new filesystem API. Run this command to upgrade:

pip install --upgrade dask dask[dataframe]

This way, Dask will automatically use pyarrow.fs under the hood without you needing to manually specify the filesystem.

3. Downgrade PyArrow (Not Recommended) #

If you can't upgrade Dask for some reason, you can revert to a PyArrow version where pyarrow.hdfs still exists (0.16.x or earlier):

pip install pyarrow==0.16.0

Note: This is a temporary workaround—old PyArrow versions may have security vulnerabilities or missing features, so upgrading Dask is the better long-term fix.

内容的提问来源于stack exchange，提问作者Santosh Kumar