inotifywait触发close_write后文件仍不可访问？上传失败排查

问题描述 #

我有一套系统：一个bash脚本每隔几秒在指定目录创建zip文件；另一个bash脚本通过inotifywait监控这些.zip文件，使用wget将其上传至Amazon S3存储桶，整体运行正常。但偶尔会出现wget上传失败，提示zip文件不存在的情况——似乎inotifywait报告文件已存在时，文件还未准备好被打开。我使用的是close_write事件来检测文件生成。

上传脚本代码 #

#!/bin/bash

# Watches for new files appearing in a zipfiles directory, and uploads them
# to AWS.

# Process a single .txt file
# $1 = filename
# $2 = the number contained in the filename

process_zip_file( )
{
    wget --no-check-certificate \
         -O /dev/null \
         --method PUT \
         --timeout=0 \
         --header 'Content-Type: application/zip' \
         --body-file=$1 \
         https://[redacted AWS endpoint]/${1}

    if [[ $? -eq 0 ]]; then
        mv $1 $UPLOADED/
    fi
}

# Wait for CLOSE_WRITE events in the data directory, and extract the results
# into an array. aline[0] is the path, [1] is the event(s), [2] is the filename

inotifywait -m -e close_write $WATCHDIR | while read -a aline; do
    fname=${aline[2]}

    # Check it is of the form zip-XYZ.zip where XYZ is a number
    if [[ $fname =~ ^zip\-([[:digit:]]*)\.zip$ ]]; then
        process_zip_file $fname ${BASH_REMATCH[1]}
    fi
done

错误日志 #

--2023-02-09 23:21:25--  https://[redacted AWS endpoint]/zip-00002331.zip
BODY data file 'zip-00002331.zip' missing: No such file or directory
--2023-02-09 23:21:33--  https://[redacted AWS endpoint]/zip-00002332.zip
Resolving [redacted AWS endpoint... [redacted IP addresses]
Connecting to [redacted AWS endpoint]|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 0 [application/json]
Saving to: '/dev/null'

     0K                                                        0.00 =0s
                                   0.00 =0s

日志显示23:21:25上传失败（提示文件不存在），约8秒后上传成功。

系统环境与zip创建逻辑 #

系统运行在Nvidia Jetson上的Yocto系统，内核版本4.9.253-l4t-r32.7，文件系统为ext4。

创建zip文件的bash脚本通过inotifywait监控C++程序生成的.txt文件，当检测到.txt文件且对应的3张.jpg文件都存在时，将这些文件打包成zip文件，代码如下：

#!/bin/bash

# Watches for new files appearing in a data directory, and once a
# .txt file and three image files exist, zips them up into a zip file.

# Process a single .txt file
# $1 = filename
# $2 = the number contained in the filename

process_txt_file( )
{
    # Check if photos are all present
    if [[ -f pp-${2}-0.jpg && -f pp-${2}-1.jpg && -f pp-${2}-2.jpg ]] ; then
        echo "Got all the parts for $2"

        # Create the zip file
        zip -r zipfiles/zip-${2}.zip \
            $1 pp-${2}-0.jpg pp-${2}-1.jpg pp-${2}-2.jpg

        if [[ $? -eq 0 ]]; then
            rm $1 pp-${2}-0.jpg pp-${2}-1.jpg pp-${2}-2.jpg
        fi
    else
        echo "Not got all the parts for $2"
    fi
}

# Wait for CLOSE_WRITE events in the data directory, and extract the results
# into an array. aline[0] is the path, [1] is the event(s), [2] is the filename

inotifywait -m -e close_write $DATA_DIR | while read -a aline; do
    fname=${aline[2]}

    # Check it is of the form pb-XYZ.txt where XYZ is a number
    if [[ $fname =~ ^pb\-([[:digit:]]*)\.txt$ ]]; then
        process_txt_file $fname ${BASH_REMATCH[1]}
    fi
done

错误发生时该脚本日志显示正常，无异常输出。

原因分析 #

问题核心是close_write事件触发时机与文件系统的延迟写入/缓存机制不匹配：

1. zip命令完成文件写入后，内核会优先将数据缓存到内存，而非立即刷写到磁盘。此时close_write事件已经触发，但文件数据还未完全落地到磁盘。
2. 上传脚本尝试用wget读取文件时，内核可能还在后台完成磁盘写入，甚至极端情况下，文件的目录项已可见，但实际数据块尚未同步完成，导致wget无法找到完整文件。
3. ext4文件系统的延迟分配、目录索引特性，也可能导致文件创建后短时间内无法被其他进程立即访问。

解决方案 #

方案1：上传前验证文件稳定性 #

修改上传脚本的process_zip_file函数，增加文件存在性校验循环，确保文件完全可用后再上传：

process_zip_file( )
{
    local fname="$1"
    # 最多等待10秒，直到文件稳定存在
    local max_wait=10
    local wait_count=0
    while [[ ! -f "$fname" && $wait_count -lt $max_wait ]]; do
        sleep 0.5
        ((wait_count++))
    done

    if [[ ! -f "$fname" ]]; then
        echo "Error: File $fname still not exists after $max_wait seconds"
        return 1
    fi

    wget --no-check-certificate \
         -O /dev/null \
         --method PUT \
         --timeout=0 \
         --header 'Content-Type: application/zip' \
         --body-file="$fname" \
         https://[redacted AWS endpoint]/${fname}

    if [[ $? -eq 0 ]]; then
        mv "$fname" "$UPLOADED/"
    fi
}

方案2：强制zip文件刷写磁盘 #

在zip创建脚本中，完成压缩后显式调用sync命令，强制内核将缓存数据刷写到磁盘，确保文件完全落地后再触发后续操作：

process_txt_file( )
{
    # Check if photos are all present
    if [[ -f pp-${2}-0.jpg && -f pp-${2}-1.jpg && -f pp-${2}-2.jpg ]] ; then
        echo "Got all the parts for $2"

        # Create the zip file
        zip -r zipfiles/zip-${2}.zip \
            $1 pp-${2}-0.jpg pp-${2}-1.jpg pp-${2}-2.jpg

        if [[ $? -eq 0 ]]; then
            # 强制刷写磁盘，确保zip文件完全落地
            sync zipfiles/zip-${2}.zip
            rm $1 pp-${2}-0.jpg pp-${2}-1.jpg pp-${2}-2.jpg
        fi
    else
        echo "Not got all the parts for $2"
    fi
}

方案3：监听create事件+等待文件大小稳定 #

改用create事件监听文件创建，然后循环检查文件大小是否稳定（连续两次检查大小一致，说明写入完成）：

inotifywait -m -e create $WATCHDIR | while read -a aline; do
    fname=${aline[2]}
    if [[ $fname =~ ^zip\-([[:digit:]]*)\.zip$ ]]; then
        # 等待文件大小稳定
        local prev_size=0
        local curr_size=$(stat -c%s "$WATCHDIR/$fname")
        while [[ $curr_size -ne $prev_size ]]; do
            sleep 0.1
            prev_size=$curr_size
            curr_size=$(stat -c%s "$WATCHDIR/$fname")
        done
        process_zip_file "$WATCHDIR/$fname" ${BASH_REMATCH[1]}
    fi
done

内容的提问来源于stack exchange，提问作者pmacfarlane