如何借助Spark编程API向需认证的外部HDFS写入数据（简单认证）

Got it, let’s walk through how to get your Spark job writing data to that external HDFS cluster—especially since your Spark runtime user doesn’t match the authenticated HDFS user you need to use.

1. Core Configuration Setup #

First, you need to tell Spark to target the external HDFS cluster and impersonate the correct authenticated user. Here are the non-negotiable configs:

• fs.defaultFS: Points to the root URI of your external HDFS cluster (matches your provided hdfs://host:port)
• hadoop.security.authentication: Set to simple to enable simple authentication mode
• HADOOP_USER_NAME: This is critical—set this to the authenticated HDFS user you’ve been given. Spark will run all HDFS operations as this user instead of its default runtime user.

2. Apply Configs (Two Common Methods) #

Option A: Pass via Spark Submit Command #

When launching your job, add these configs as --conf parameters:

spark-submit \
  --conf spark.hadoop.fs.defaultFS=hdfs://host:port \
  --conf spark.hadoop.hadoop.security.authentication=simple \
  --conf spark.hadoop.HADOOP_USER_NAME=your_authenticated_hdfs_user \
  --class your.main.application.class \
  your-spark-app.jar

Option B: Set Directly in Code #

If you prefer configuring within your application, here’s how to do it in Scala and Python:

Scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("WriteToExternalHDFS")
  .config("spark.hadoop.fs.defaultFS", "hdfs://host:port")
  .config("spark.hadoop.hadoop.security.authentication", "simple")
  .config("spark.hadoop.HADOOP_USER_NAME", "your_authenticated_hdfs_user")
  .getOrCreate()

Python

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("WriteToExternalHDFS") \
    .config("spark.hadoop.fs.defaultFS", "hdfs://host:port") \
    .config("spark.hadoop.hadoop.security.authentication", "simple") \
    .config("spark.hadoop.HADOOP_USER_NAME", "your_authenticated_hdfs_user") \
    .getOrCreate()

3. Write Data to the Target Path #

Once your Spark session is set up, writing data is straightforward. Use your target path hdfs://host:port/loc and specify your desired file format:

Example (Writing a DataFrame): #

Scala

// Assume you have a DataFrame named `df` ready to write
df.write
  .format("parquet") // Replace with your format (csv, json, etc.)
  .mode("overwrite") // Choose mode: overwrite/append/ignore/errorifexists
  .save("hdfs://host:port/loc")

Python

# Assume you have a DataFrame named `df` ready to write
df.write \
    .format("parquet")  # Replace with your format (csv, json, etc.)
    .mode("overwrite")  # Choose mode: overwrite/append/ignore/errorifexists
    .save("hdfs://host:port/loc")

Quick Troubleshooting Tips #

• Permission Checks: Make sure the authenticated HDFS user has write access to hdfs://host:port/loc—you can verify this by running hdfs dfs -mkdir -p hdfs://host:port/loc directly as that user.
• Network Access: Confirm your Spark cluster can reach the external HDFS host and port (no firewalls blocking the connection).
• Config Overrides: If your Spark cluster has a default HDFS config, double-check that the external cluster settings are overriding the defaults correctly.

内容的提问来源于stack exchange，提问作者A.G.