如何将hadoop fs -ls命令输出保存至HDFS目录？

Hey there! Let's fix this issue—your current commands aren't working because the > redirect writes output to your local file system, not HDFS. Here are a few straightforward ways to save the hadoop fs -ls / results directly to an HDFS directory:

Method 1: Local File Intermediate (Easy to Troubleshoot) #

First, save the output to a local file, then upload it to HDFS:

# Step 1: Write listing to a local file
hadoop fs -ls / > local_lsResult.txt

# Step 2: Upload the local file to your target HDFS directory (replace /dir1 with your desired path)
hadoop fs -put local_lsResult.txt /dir1/lsResult.txt

# Optional: Delete the local file once uploaded
rm local_lsResult.txt

Method 2: Direct Pipe to HDFS (No Local File Needed) #

Use a pipe to send the command output directly to HDFS without creating a local file. The - in hadoop fs -put - tells Hadoop to read from standard input:

hadoop fs -ls / | hadoop fs -put - /dir1/lsResult.txt

Method 3: Save to Both Local and HDFS (With tee) #

If you want a local backup and the HDFS file, use tee to split the output:

hadoop fs -ls / | tee local_lsResult.txt | hadoop fs -put - /dir1/lsResult.txt

Important Notes: #

• Make sure you have write permissions on the target HDFS directory (e.g., /dir1 in the examples above). If you get a permission error, try using a directory you own like /user/hadoop/lsResult.txt.
• Replace /dir1/lsResult.txt with your actual desired HDFS path (e.g., /lsResult.txt to save directly in the HDFS root).

内容的提问来源于stack exchange，提问作者Shinghim LEE