为何解压Shell脚本在Dev与Prod环境下表现存在差异？

Let's break down the root causes of the inconsistent behavior between your Dev and Prod environments, along with actionable fixes:

Key Reasons for the Difference #

1. Shell Syntax Error (Missing Semicolon) #

Your script has a syntax issue in the for loop:

for f in $files do

The correct syntax for a one-line for loop requires a semicolon between the loop list and do:

for f in $files; do

Some shells (like bash in your Dev cluster) are more lenient and auto-correct this mistake, splitting the loop variable properly. But stricter shell environments (like sh or bash in strict mode on Prod) misparse this line, treating all your GZ file paths as a single value for f—which explains why you see all three paths printed in one line on Prod.

2. IFS (Internal Field Separator) Mismatch #

The for f in $files pattern relies on the shell's IFS to split the string into individual items:

• In Dev, the default IFS is used (including spaces, newlines, and tabs), so the space-separated paths from your tr '\n' ' ' command get split correctly into separate loop iterations.
• In Prod, the IFS might have been modified (e.g., set to only newlines via IFS=$'\n' in a profile script). This means the shell won't split on spaces, so all paths are treated as a single value for f.

3. Fragile Hadoop Output Parsing #

Your method of extracting filenames with hadoop fs -ls $f | awk -F '/' '{print $NF}' | head -1 is fragile:

• If f contains multiple paths (as it does on Prod), hadoop fs -ls outputs multiple lines, and head -1 only grabs the first filename—leading to empty values when subsequent commands try to use ${filename%.*}.
• Different Hadoop versions on Dev vs Prod might have slight variations in hadoop fs -ls output format, which can break the awk parsing.

Fixed Script (More Reliable) #

Replace your original script with this version, which addresses all the above issues:

#!/bin/bash
# Reset IFS to default to avoid splitting issues
IFS=$' \t\n'

output_path="s3://output"
unziped_dir="s3://2019-01-03"

# Use while loop to read each file path directly (no tr needed)
hadoop fs -ls "$output_path/" | awk '{print $NF}' | grep .gz$ | while read -r gz_file; do
    echo "available files are: $gz_file"
    # Use basename to safely extract the filename (more reliable than parsing ls output)
    filename=$(basename "$gz_file")
    # Unzip and write to target directory
    hdfs dfs -cat "$gz_file" | gzip -d | hdfs dfs -put - "$unziped_dir/${filename%.*}"
    echo "unziped file names: ${filename%.*}"
done

Additional Improvements #

• Quoted Variables: Wrapping variables like "$gz_file" and "$output_path" prevents issues if paths ever contain spaces or special characters.
• Avoid Parsing ls Output: Using basename is a safer way to get filenames than parsing hadoop fs -ls output, which can vary between Hadoop versions.
• Explicit Shell Shebang: Adding #!/bin/bash ensures the script runs with bash (more consistent across EMR clusters) instead of the default sh.

内容的提问来源于stack exchange，提问作者z_1_p