Python 2.7 multiprocessing.Pool删除文件触发WindowsError[32]求助

背景与执行流程 #

在Python 2.7环境下（因框架兼容性限制），执行以下处理流程：

• 读取ID_something.py文件内容后关闭文件
• 创建out_something.py用于存储输出
• 从每行提取ID字符串，无ID则跳过后续步骤
• 利用multiprocessing.Pool多进程批量检索ID在其他模块文件中的出现次数
• 将结果写入输出文件，处理完所有ID后关闭文件
• 若输出文件为空则尝试删除该文件

相关代码 #

#open ID file
id_file = open(id_prefix+filename+".py","r") #open the ID file
id_lines = id_file.readlines() #read its contents
id_file.close()
#write out file with the results
if os.path.exists(out_prefix+filename+".py"):
    print "ERROR: file "+out_prefix+filename+".py already exists! Program must be stopped." #user must have created the file while the program was running! (or deletion of the file failed)
    exit(1) #error: out file would overwrite an existing file
out_file = open(out_prefix+filename+".py","w") #open the out file
print "Searching for IDs from "+(id_prefix if use_subfolders else "")+id_filename+"..."
line_break = get_file_comment(filename) #this is what we add before the next line, where each line is the id and its occurences, so before the 1st line print a general comment about the whole file (or just make it an empty string)
#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#!#
#execute searches for IDs in the file
#prepare a list of arguments
list_of_args = [(id_line,filename,temp_path,temp_match_prefix,temp_prefix) for id_line in id_lines]
#assign work to a pool of worker processes
out_lines = multiprocessing.Pool().map(process_id_occurrences,list_of_args)
for out_line in out_lines:
    if not out_line=="":
        out_file.write(line_break+out_line)
        line_break = "\n" #will be added before next line
#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#@#
out_file.close()
if os.path.getsize(out_prefix+filename+".py")==0: #we can't refer to out_file variable, we have to refer to the actual file on the disk
    overwrite_prev_prnt = True
    os.remove(out_prefix+filename+".py") #<-- THIS LINE CAUSES AN ERROR

遇到的问题 #

执行os.remove删除空文件时触发WindowsError: [Error 32]，提示文件被其他进程占用。已调用out_file.close()，尝试手动控制Pool.close()仍无效；添加pool.join()触发AssertionError；延迟删除、循环重试均无法解决，确认文件被多个python.exe进程锁定。

问题根源 #

1. 临时Pool对象未正确回收子进程：代码中直接使用multiprocessing.Pool().map(...)创建临时Pool对象，调用map后虽会阻塞等待任务完成，但Pool的子进程不会立即终止，Windows系统下这些子进程可能仍持有相关资源，导致文件被锁定。
2. 未显式关闭并等待Pool终止：没有保存Pool对象引用，无法调用close()和join()确保所有子进程完全退出，子进程残留的文件句柄或资源会占用目标文件。
3. 文件句柄管理不严谨：手动调用out_file.close()的可靠性不如with语句的自动资源管理，若中间出现异常可能导致文件未正确关闭；此外需检查process_id_occurrences函数是否存在未正确关闭的文件句柄，尤其是是否意外打开了输出文件。

解决方法 #

1. 显式管理Pool生命周期 #

保存Pool对象引用，在map执行完成后显式关闭Pool并等待子进程全部终止：

# 创建Pool对象并保存引用
pool = multiprocessing.Pool()
out_lines = pool.map(process_id_occurrences, list_of_args)
# 禁止提交新任务，等待所有子进程完成
pool.close()
pool.join()

此操作确保所有子进程完全退出，释放占用的资源和文件句柄。

2. 使用with语句管理文件 #

用with语句替代手动open/close，确保文件无论是否发生异常都会被正确关闭：

# 替换原有的out_file打开逻辑
with open(out_prefix+filename+".py", "w") as out_file:
    print "Searching for IDs from "+(id_prefix if use_subfolders else "")+id_filename+"..."
    line_break = get_file_comment(filename)
    # 执行多进程检索
    pool = multiprocessing.Pool()
    out_lines = pool.map(process_id_occurrences, list_of_args)
    pool.close()
    pool.join()
    # 写入结果
    for out_line in out_lines:
        if not out_line == "":
            out_file.write(line_break+out_line)
            line_break = "\n"

3. 检查process_id_occurrences函数 #

确保该函数中所有打开的文件都通过with语句或显式close()关闭，避免子进程残留未释放的文件句柄，尤其是确认没有意外打开输出文件out_something.py。

4. 重试删除（可选） #

若上述方法仍有偶发问题，可在删除时添加重试逻辑，给系统足够时间释放文件锁：

import time

if os.path.getsize(out_prefix+filename+".py") == 0:
    overwrite_prev_prnt = True
    max_retries = 3
    retry_count = 0
    while retry_count < max_retries:
        try:
            os.remove(out_prefix+filename+".py")
            break
        except WindowsError:
            retry_count += 1
            time.sleep(0.5)
    else:
        print "Failed to delete empty file after multiple retries"

内容的提问来源于stack exchange，提问作者Artur