Windows环境下通过SparkR访问HDFS中的Parquet文件

It looks like you’re on the right track, but there are a few common Windows-specific pitfalls and configuration gaps in your setup that might be blocking access to the remote cluster. Let’s walk through fixing them step by step:

1. Fix the HADOOP_HOME Path (Critical for Windows) #

Your current HADOOP_HOME uses a Unix-style path (/opt/hadoop-2.9.0), which won’t work on Windows. You need to:

• Set it to the full Windows path of your Hadoop installation (ensure you have Windows-compatible Hadoop binaries with winutils.exe in the bin folder)
• Use escaped backslashes (\\) in R strings to avoid path errors

Example correction:

Sys.setenv(
  SPARK_HOME = "C:\\Users\\me\\Hadoop\\spark-2.3.0-bin-hadoop2.7",
  HADOOP_HOME = "C:\\Users\\me\\Hadoop\\hadoop-2.9.0",  # Windows path here
  SPARK_HOME_VERSION = "2.3.0"
)

2. Add HDFS Configuration to Your Spark Session #

When connecting to a remote cluster, Spark needs explicit guidance to locate the Hadoop NameNode. Add this key config to your sparkR.session call:

sc <- sparkR.session(
  enableHiveSupport = FALSE,
  master = "spark://10.123.45.67:7077",
  sparkConfig = list(
    spark.driver.memory = "2g",
    spark.hadoop.fs.defaultFS = "hdfs://10.123.45.67:9000"  # Replace with your cluster's NameNode port (usually 9000)
  )
)

3. Read Parquet Files with the Full HDFS Path #

Don’t use partial paths — specify the complete HDFS path to your Parquet directory or files:

# Replace with your actual HDFS path to the Parquet data
patient <- read.parquet("hdfs://10.123.45.67:9000/user/your_username/patient_data.parquet")

Additional Troubleshooting Tips #

• Firewall Checks: Ensure your Windows machine can reach the remote Spark master (port 7077) and HDFS NameNode (port 9000) — verify both your local firewall and the cluster’s network rules.
• Winutils.exe: Windows requires this Hadoop utility to handle file system operations. Download the version matching your Hadoop release and place it in HADOOP_HOME/bin.
• Version Compatibility: Confirm your local Spark version (2.3.0) matches the version running on the remote cluster — mismatched versions often cause connection failures.
• Log Review: If you hit errors, check the Spark driver logs (visible in your RStudio console or temporary directory) for specific messages that can pinpoint the issue.

内容的提问来源于stack exchange，提问作者N.J.