如何在边缘节点用Scala读取文本文件？解决文件未找到报错

Let's break down what's going wrong here and fix it step by step:

Key Issues in Your Code #

• Incorrectly converting Spark RDD to String: When you call sc.textFile(...), you get an RDD[String] (a distributed collection of file lines), not a file path string. Calling .toString() on this RDD only returns a debug identifier like org.apache.spark.rdd.MapPartitionsRDD@xxxx—not the actual file path your readFile function expects. This is the main reason for the "file not found" error.
• Malformed file URI: The file path should use file:/// (three slashes after file:) for absolute local paths. Your current path file://home//... is missing a slash, which breaks path resolution.
• Unnecessary mix of Spark and local file reading: If you're just reading a local file on the edge node, you don't need Spark at all—direct local file reading is simpler and more efficient.

Solution 1: Read Local File Directly (No Spark Needed) #

If your goal is to read a local text file on the edge node, use Scala's built-in Source class directly with the correct absolute path:

import scala.io.Source

object LocalFileReader {
  def main(args: Array[String]): Unit = {
    // Use absolute path (you can also use "file:///home/..." if preferred)
    val srcFilePath = "/home/viji.palanisamy/dev/kpi_library/EDI/Prof_test1"
    readFile(srcFilePath)
  }

  def readFile(filename: String): Unit = {
    // Use try-finally to ensure the file resource is closed properly
    val bufferedSource = Source.fromFile(filename)
    try {
      // Actually read and print the file content (not just the source object)
      println("File content:")
      bufferedSource.getLines().foreach(println)
    } finally {
      bufferedSource.close()
    }
  }
}

Solution 2: Use Spark for Distributed File Reading #

If you need to process the file with Spark (e.g., distributed processing across nodes), let Spark handle the file reading directly—you don't need your custom readFile function:

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

object SparkFileProcessor {
  def main(args: Array[String]): Unit = {
    // Initialize Spark context for local execution
    val conf = new SparkConf()
      .setAppName("EdgeNodeFileReader")
      .setMaster("local[*]") // Use all available cores on the edge node
    val sc = new SparkContext(conf)

    // Correct file URI with triple slash for absolute path
    val fileRDD = sc.textFile("file:///home/viji.palanisamy/dev/kpi_library/EDI/Prof_test1")

    // Example: Process and print file content
    println("Spark-processed file content:")
    fileRDD.foreach(println)

    // Always stop the Spark context when done
    sc.stop()
  }
}

Additional Troubleshooting Checks #

• Verify file permissions: Ensure the user viji.palanisamy has read access to the file and all parent directories. Run this command to check:

  ls -l /home/viji.palanisamy/dev/kpi_library/EDI/Prof_test1
  

• Double-check the path: Typos in directory or file names are a common culprit—copy-paste the path from a terminal to avoid mistakes.

内容的提问来源于stack exchange，提问作者vijay