Got it, let's break down how to tackle this problem—since your date=YYYYMMDD directory names don't reflect the actual last modified time of files inside them, we need to go straight to the file-level metadata to get accurate results.

核心思路 #

Instead of relying on directory names to judge freshness, we'll fetch the actual last modified time of every file in your target HDFS path, sort those files by this timestamp, and pick the most recent ones.

具体步骤 #

1. 递归获取文件的修改时间与路径

Use hdfs dfs -stat to output each file's last modified time and full HDFS path. This command gives you precise metadata that isn't tied to the directory's name:

# 递归遍历所有子目录，输出格式为「修改时间 文件路径」
hdfs dfs -stat "%y %n" /your/target/hdfs/path/**

• %y: Prints the last modification time in YYYY-MM-DD HH:MM:SS ±HHMM format (includes timezone)
• %n: Prints the full HDFS path of the file
• **: Recursively matches all subdirectories (note: if your shell doesn't support this, enable globstar first with shopt -s globstar for bash)

If ** doesn't work for you, use hdfs dfs -find for reliable recursive lookup:

hdfs dfs -find /your/target/hdfs/path -type f -exec stat "%y %n" {} \;

2. 按修改时间倒序排序，筛选最新文件

Pipe the output to sort -r (reverse sort, so newest entries come first) and head to grab the top N latest files:

# 获取最新的10个文件
hdfs dfs -stat "%y %n" /your/target/hdfs/path/** | sort -r | head -10

3. 提取纯文件路径（可选）

If you only need the file paths (not the timestamp), use awk to extract the last field from the output:

hdfs dfs -stat "%y %n" /your/target/hdfs/path/** | sort -r | head -5 | awk '{print $NF}'

额外提示 #

• If you're dealing with a massive number of files, this command might take time—consider running it during off-peak hours, or use tools like Spark to scan HDFS metadata more efficiently for large-scale datasets.
• For automation, wrap this logic in a shell script that saves the latest file path to a variable, which you can then use in downstream data processing jobs.

内容的提问来源于stack exchange，提问作者knowone