Scala读取HDFS文件创建RDD遇报错求助

Hey there, let's break down the issues in your code that are likely causing those frustrating errors when trying to load files from HDFS:

1. You're not assigning the SparkContext to sc #

Looking at your code snippet, you create a SparkContext but don't store it in the sc variable you later reference:

new SparkContext(conf) // This creates a SparkContext but doesn't save it to `sc`
val rdd = sc.textFile("/test/dir/*") // `sc` is undefined here—no wonder it errors!

Fix this by assigning the SparkContext to sc explicitly:

val sc = new SparkContext(conf)

2. Your master URL is invalid #

The setMaster("master") line won't work—Spark needs a valid master address:

• For local testing (great for debugging), use setMaster("local[*]") to use all your machine's cores, or local for a single core.
• For a cluster environment, use the actual master URL like spark://your-cluster-master:7077.

3. HDFS path mismatches or missing configuration #

You mentioned your HDFS path is hdfs/test/dir/text.txt, but your code uses /test/dir/*. Here's what to check:

• If you're using a fully qualified HDFS path, it should start with hdfs:// followed by your namenode's address and port, e.g., hdfs://your-namenode:9000/test/dir/*.
• If your Spark cluster is configured to use HDFS as the default filesystem, /test/dir/* works—but first verify the directory exists with the HDFS command: hdfs dfs -ls /test/dir.
• Also, ensure your Spark app has access to Hadoop's config files (core-site.xml, hdfs-site.xml)—either place them in Spark's conf folder or set the HADOOP_CONF_DIR environment variable when running your app.

Pro Tip: Use SparkSession (Modern Spark Best Practice) #

While SparkContext works, newer Spark versions recommend using SparkSession—it combines SparkContext and SQLContext into one easy-to-use entry point:

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("training")
  .master("local[*]") // Adjust this for your cluster
  .getOrCreate()

// Load files using the SparkContext attached to SparkSession
val rdd = spark.sparkContext.textFile("hdfs://your-namenode:9000/test/dir/*")

内容的提问来源于stack exchange，提问作者user6203336