R编程新手求助：配置Hadoop环境后to.dfs执行报错

Hey there! Let's figure out why you're getting that "Not a valid jar" error when using to.dfs in R. I’ve dealt with this exact issue before when working with rhdfs and rmr2, so here’s a step-by-step breakdown to fix it:

1. Double-Check Your HADOOP_STREAMING Path #

The most common culprit here is a wrong path to the hadoop-streaming jar—let’s start here:

• First, confirm the jar actually exists at the path you set. Run this in your terminal:

  ls /usr/local/hadoop/share/hadoop/tool/lib/hadoop-streaming-2.6.0.jar
  

  If you get a "No such file or directory" error, you probably made a tiny typo: Hadoop streaming jars are almost always in share/hadoop/tools/lib (plural tools, not singular tool). To find the exact path, run this search command:

  find /usr/local/hadoop -name "hadoop-streaming*.jar"
  

  Use the full path from the search result to update your HADOOP_STREAMING environment variable.

2. Verify the Jar File Isn’t Corrupted #

If the path checks out, make sure the jar itself isn’t damaged. Run this terminal command to inspect the jar’s contents:

jar tf /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar

If you get errors like "invalid jar file", you’ll need to replace the jar with a valid copy from your Hadoop distribution (re-download it or grab it from a working Hadoop setup).

3. Reinitialize Your R Environment #

Sometimes fixing the path requires a fresh start. Restart your R session, reset the environment variables with the correct jar path, then reload the libraries and test again:

# Reset environment variables with the corrected jar path
Sys.setenv("HADOOP_CMD"="/usr/local/hadoop/bin/hadoop")
Sys.setenv("HADOOP_CONF"="/usr/local/hadoop/conf")
# Replace this path with the one you found from the 'find' command
Sys.setenv("HADOOP_STREAMING"="/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar")

library(rhdfs)
hdfs.init()
library(rmr2)
library(plyrmr)

# Test with a simple to.dfs call
sample <- 1:500
to.dfs(sample, "/user/hduser/test_sample")

4. Check Version Compatibility #

Make sure the hadoop-streaming jar version matches your Hadoop cluster version. If your cluster is running a different version than 2.6.0, a mismatched jar can trigger this error. Confirm your Hadoop version with:

hadoop version

Then use the streaming jar that corresponds to that exact version.

内容的提问来源于stack exchange，提问作者user8915561