使用GNU Parallel --line-buffer仍遇并发输出异常：行合并或截断

问题描述 #

我编写了一个用awk处理二级结构数据并将结果追加至最终输出文件的函数，尽管已在GNU Parallel中使用--line-buffer参数，输出文件仍偶尔出现行合并或截断的情况：

4 GLN A 447 C 1 GLN A 1 T

或是截断的行：

4 GLN A 447

经排查是多个进程同时写入文件导致的冲突，相关代码片段如下：

calculate_secondary_structure() {
    frame_counter=$1
    # process only chain A, as it is polyQ
    ${stride_path}/stride ${pdbs_dir}/pdb_for_cluster${frame_counter}.pdb -ca -fsecondary_structure${frame_counter}.txt
    rm ${pdbs_dir}/pdb_for_cluster${frame_counter}.pdb

    awk -v frame_counter="$frame_counter" '
    BEGIN { OFS="          " } # 10 spaces as a separator
    /^ASG/ {
        residue_name = substr($0, 6, 3)
        chain_name = substr($0, 10, 1)
        residue_number = substr($0, 17, 4)
        ss_code = substr($0, 25, 1)

        # Print the frame number followed by the extracted fields with 10 spaces between them
        printf "%-10s %-10s %-10s %-10s %-10s\n", frame_counter, residue_name, chain_name, residue_number, ss_code
    }' "secondary_structure${frame_counter}.txt" >> ${final_output_file} # Append directly to the final file

    rm secondary_structure${frame_counter}.txt
}

export -f calculate_secondary_structure
seq 1 ${number_of_frames} | parallel --bar --line-buffer --block 1k --round-robin -j192 calculate_secondary_structure {}

Parallel版本：GNU parallel 20240822

问题根源 #

--line-buffer仅控制GNU Parallel自身的输出缓冲，但函数内是让awk直接将内容追加到最终文件。此时多个awk进程同时写入同一文件，操作系统无法保证行写入的原子性，因此会出现行重叠或截断。

解决方法 #

方案1：让Parallel统一处理输出写入 #

取消函数内的直接追加操作，让awk将结果输出到stdout，由Parallel收集所有进程的输出后统一写入文件。这样--line-buffer才能真正发挥作用，避免多进程写冲突：

calculate_secondary_structure() {
    frame_counter=$1
    ${stride_path}/stride ${pdbs_dir}/pdb_for_cluster${frame_counter}.pdb -ca -fsecondary_structure${frame_counter}.txt
    rm ${pdbs_dir}/pdb_for_cluster${frame_counter}.pdb

    awk -v frame_counter="$frame_counter" '
    BEGIN { OFS="          " }
    /^ASG/ {
        residue_name = substr($0, 6, 3)
        chain_name = substr($0, 10, 1)
        residue_number = substr($0, 17, 4)
        ss_code = substr($0, 25, 1)
        printf "%-10s %-10s %-10s %-10s %-10s\n", frame_counter, residue_name, chain_name, residue_number, ss_code
    }' "secondary_structure${frame_counter}.txt"

    rm secondary_structure${frame_counter}.txt
}

export -f calculate_secondary_structure
seq 1 ${number_of_frames} | parallel --bar --line-buffer --block 1k --round-robin -j192 calculate_secondary_structure {} >> ${final_output_file}

方案2：使用临时文件合并输出 #

每个进程先将结果写入独立的临时文件，待所有进程执行完成后，再将所有临时文件合并到最终输出文件。这种方式完全避免并行写冲突：

calculate_secondary_structure() {
    frame_counter=$1
    temp_file="temp_ss_${frame_counter}.txt"
    ${stride_path}/stride ${pdbs_dir}/pdb_for_cluster${frame_counter}.pdb -ca -fsecondary_structure${frame_counter}.txt
    rm ${pdbs_dir}/pdb_for_cluster${frame_counter}.pdb

    awk -v frame_counter="$frame_counter" '
    BEGIN { OFS="          " }
    /^ASG/ {
        residue_name = substr($0, 6, 3)
        chain_name = substr($0, 10, 1)
        residue_number = substr($0, 17, 4)
        ss_code = substr($0, 25, 1)
        printf "%-10s %-10s %-10s %-10s %-10s\n", frame_counter, residue_name, chain_name, residue_number, ss_code
    }' "secondary_structure${frame_counter}.txt" > "$temp_file"

    rm secondary_structure${frame_counter}.txt
}

export -f calculate_secondary_structure
seq 1 ${number_of_frames} | parallel --bar --block 1k --round-robin -j192 calculate_secondary_structure {}
# 合并临时文件（若需按帧顺序合并，可改为遍历seq顺序的临时文件）
cat temp_ss_*.txt >> ${final_output_file}
rm temp_ss_*.txt

方案3：使用文件锁保证写入原子性 #

借助flock工具，在写入文件前加排他锁，确保同一时间只有一个进程写入。此方式会让写操作串行化，可能影响并行效率，但能严格保证写入的原子性：

calculate_secondary_structure() {
    frame_counter=$1
    ${stride_path}/stride ${pdbs_dir}/pdb_for_cluster${frame_counter}.pdb -ca -fsecondary_structure${frame_counter}.txt
    rm ${pdbs_dir}/pdb_for_cluster${frame_counter}.pdb

    awk -v frame_counter="$frame_counter" '
    BEGIN { OFS="          " }
    /^ASG/ {
        residue_name = substr($0, 6, 3)
        chain_name = substr($0, 10, 1)
        residue_number = substr($0, 17, 4)
        ss_code = substr($0, 25, 1)
        printf "%-10s %-10s %-10s %-10s %-10s\n", frame_counter, residue_name, chain_name, residue_number, ss_code
    }' "secondary_structure${frame_counter}.txt" | flock -x ${final_output_file} -c 'cat >> '$final_output_file''

    rm secondary_structure${frame_counter}.txt
}

export -f calculate_secondary_structure
seq 1 ${number_of_frames} | parallel --bar --line-buffer --block 1k --round-robin -j192 calculate_secondary_structure {}

总结 #

优先选择方案1，这种方式既保留了并行处理的效率，又能通过Parallel统一管理输出避免写冲突，是最简洁高效的解决办法。

内容的提问来源于stack exchange，提问作者Adupa Vasista