运行Hadoop示例ShowFileStatusTest时遇文件不存在错误求助

Hey there! Let's work through why your ShowFileStatusTest's fileStatusForFile function keeps failing with that "File does not exist" error. As someone who's stumbled through early Hadoop testing hurdles, I get how frustrating this can be—let's break down the most likely fixes step by step:

1. Ensure the Test File is Actually Created First #

Unit tests don't magically generate the files they need to test! You'll need to manually create the target file in either the local filesystem or HDFS before running the test:

• Local filesystem: Use the command touch /absolute/path/to/your/testfile.txt to create an empty test file, or copy an existing file with cp source.txt /target/path/testfile.txt.
• HDFS: First create the parent directory if it doesn't exist: hdfs dfs -mkdir -p /hdfs/test/path, then upload a file with hdfs dfs -put local-testfile.txt /hdfs/test/path/testfile.txt.
• Critical check: Double that the path in your test code matches the path you created exactly—HDFS paths are case-sensitive, so even a tiny typo (like TestFile.txt vs testfile.txt) will cause this error.

2. Verify File System Configuration in Your Code #

Make sure your test is pointing to the right file system:

• If testing HDFS: Ensure your code is loading the correct Hadoop configuration files (core-site.xml, hdfs-site.xml) or explicitly specifying the HDFS URI. For example:

  Configuration conf = new Configuration();
  FileSystem fs = FileSystem.get(new URI("hdfs://localhost:9000"), conf);
  

  Without this, your test might default to the local filesystem even if you intend to test HDFS.
• If testing the local filesystem: Use absolute paths instead of relative paths to avoid confusion about the working directory where the test runs. For example, use /home/youruser/testfile.txt instead of testfile.txt.

3. Check File System Permissions #

Permissions are a common culprit here:

• HDFS: Run hdfs dfs -ls /hdfs/test/path to confirm the file exists and your user has read access. If permissions are restricted, adjust them with hdfs dfs -chmod 755 /hdfs/test/path (or set appropriate ownership with hdfs dfs -chown youruser:yourgroup /hdfs/test/path).
• Local filesystem: Use ls -l /absolute/path/to/testfile.txt to check that the user running the test has read permissions. If not, run chmod 644 /absolute/path/to/testfile.txt to grant read access.

4. Audit Test Code Logic for Accidental File Deletion #

Check if your test setup/teardown is removing the file before the test runs:

• Look for @Before or @After annotations in your test class—some tests include cleanup logic that deletes files, which might be targeting your test file by mistake.
• Confirm that the FileSystem instance in fileStatusForFile is the same one you intended to use. For example, if you're trying to test HDFS but accidentally initialized a local FileSystem instance, it won't find the file in HDFS.

5. Get Full Error Logs for Better Debugging #

The partial log you shared isn't enough to pinpoint the exact issue. To dig deeper:

• Print the full stack trace of the exception—this will show exactly where in the code the error is thrown (e.g., during getFileStatus() or when resolving the path).
• Add debug prints to your test code, like:

  System.out.println("Using file system: " + fs.getUri());
  System.out.println("Checking file path: " + filePath);
  

  This will confirm that your test is looking in the right place.

内容的提问来源于stack exchange，提问作者Kachuan Lin