运行spark-submit时出现FileSystem异常，寻求技术帮助

Hey Nitin, let's work through this spark-submit issue you're hitting! First, let's break down what's going on:

核心问题分析 #

The No FileSystem for scheme: C error is the real blocker here — the native Hadoop library warning is just a minor nuisance (it won't stop your job from running, though we can address it too if you want).

Hadoop's FileSystem API doesn't recognize raw Windows drive letters like C: as a valid scheme. It expects explicit scheme identifiers (like file:// for local filesystems or hdfs:// for HDFS) to know which filesystem handler to use.

解决方案 #

1. 修正文件路径格式 #

Change any paths starting with C: to use the file:// scheme. For example:

• Instead of C:\your\input\path, use file:///C:/your/input/path
• If you're passing paths via command line or in your code, make sure to format them this way.

Here's how your spark-submit command might look with the corrected path:

spark-submit --class your.main.Class your-app.jar --input file:///C:/data/input --output file:///C:/data/output

2. 显式指定本地文件系统实现（如果路径修正后仍有问题） #

Add this configuration to your spark-submit command to force Hadoop to use the local filesystem handler:

spark-submit --conf spark.hadoop.fs.file.impl=org.apache.hadoop.fs.LocalFileSystem --class your.main.Class your-app.jar

3. 解决Native Hadoop Library警告（可选） #

If you want to get rid of the Unable to load native-hadoop library warning:

• Download the Windows-compatible Hadoop native binaries (match your Hadoop version).
• Set the HADOOP_HOME environment variable to your Hadoop installation directory.
• Add %HADOOP_HOME%\bin to your system PATH.
• Or, pass the library path directly in spark-submit:

  spark-submit --conf spark.driver.extraLibraryPath=C:\hadoop\bin --conf spark.executor.extraLibraryPath=C:\hadoop\bin --class your.main.Class your-app.jar
  

内容的提问来源于stack exchange，提问作者Nitin Gupta