使用ThreadPool多线程下载图片遇FileNotFoundError：Ubuntu异常Windows正常

代码在Python 3.11 + Windows 11环境下运行正常，但在Python 3.11 + Ubuntu 20.04环境中执行时抛出FileNotFoundError。代码功能是通过线程池下载图片到指定目录，目标目录/media/hdd/img/存在且权限为755，无法定位问题。

from multiprocessing.pool import ThreadPool
import os
import requests

list2 = [('/media/hdd/img/tt24949716.jpg', 'https://www.cdnzone.org/uploads/2023/01/29/Devil-Beneath-2023-Cover.jpg'), 
('/media/hdd/img/tt11231356.jpg', 'https://www.cdnzone.org/uploads/2023/01/29/Beneath-the-Green-2022-Cover.jpg'), 
('/media/hdd/img/tt13103732.jpg', 'https://www.cdnzone.org/uploads/2023/01/29/The-Offering-2022-Cover.jpg'), 
('/media/hdd/img/tt14826022.jpg', 'https://www.cdnzone.org/uploads/2023/01/29/You-People-2023-Cover.jpg'), 
('/media/hdd/img/tt18252340.jpg', 'https://www.cdnzone.org/uploads/2023/01/29/Carnifex-2022-Cover.jpg')]


def download_images(list):
    results = ThreadPool(8).imap_unordered(fetch_url, list)
    for path in results:
        print(path)

def fetch_url(entry):
    path, uri = entry
    if not os.path.exists(path):
        r = requests.get(uri, stream=True)
        if r.status_code == 200:
            with open(path, 'wb') as f:
                for chunk in r:
                    f.write(chunk)
    return path

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in download_images
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 930, in __init__
  File "/usr/lib/python3.11/multiprocessing/pool.py", line 196, in __init__
  File "/usr/lib/python3.11/multiprocessing/context.py", line 113, in SimpleQueue
  File "/usr/lib/python3.11/multiprocessing/queues.py", line 341, in __init__
  File "/usr/lib/python3.11/multiprocessing/context.py", line 68, in Lock
  File "/usr/lib/python3.11/multiprocessing/synchronize.py", line 162, in __init__
  File "/usr/lib/python3.11/multiprocessing/synchronize.py", line 57, in __init__
FileNotFoundError: [Errno 2] No such file or directory

这个报错并非指向图片存储路径，而是来自Python multiprocessing模块内部的同步锁创建失败，常见原因有两个：

• 交互式环境（如Python Shell）中使用multiprocessing.pool.ThreadPool时，Ubuntu系统的IPC机制依赖/tmp目录，若该目录权限异常（非1777），会导致锁文件无法创建。
• Python 3.11在Ubuntu环境下的multiprocessing线程池实现，在交互式场景下存在兼容性问题。

方案1：改用concurrent.futures.ThreadPoolExecutor #

concurrent.futures是更上层的线程池API，跨平台和交互式环境兼容性更好，修改后的代码如下：

import os
import requests
from concurrent.futures import ThreadPoolExecutor

list2 = [('/media/hdd/img/tt24949716.jpg', 'https://www.cdnzone.org/uploads/2023/01/29/Devil-Beneath-2023-Cover.jpg'), 
('/media/hdd/img/tt11231356.jpg', 'https://www.cdnzone.org/uploads/2023/01/29/Beneath-the-Green-2022-Cover.jpg'), 
('/media/hdd/img/tt13103732.jpg', 'https://www.cdnzone.org/uploads/2023/01/29/The-Offering-2022-Cover.jpg'), 
('/media/hdd/img/tt14826022.jpg', 'https://www.cdnzone.org/uploads/2023/01/29/You-People-2023-Cover.jpg'), 
('/media/hdd/img/tt18252340.jpg', 'https://www.cdnzone.org/uploads/2023/01/29/Carnifex-2022-Cover.jpg')]

def fetch_url(entry):
    path, uri = entry
    if not os.path.exists(path):
        r = requests.get(uri, stream=True)
        if r.status_code == 200:
            with open(path, 'wb') as f:
                for chunk in r:
                    f.write(chunk)
    return path

def download_images(img_list):
    with ThreadPoolExecutor(max_workers=8) as executor:
        for path in executor.map(fetch_url, img_list):
            print(path)

download_images(list2)

方案2：修复/tmp目录权限 #

在Ubuntu终端执行以下命令，确保/tmp目录权限符合要求：

sudo chmod 1777 /tmp

执行后重新运行代码即可。

方案3：保存为脚本文件执行 #

不要在交互式Python Shell中运行，将代码保存为.py文件（比如download_imgs.py），然后在终端执行：

python3 download_imgs.py

内容的提问来源于stack exchange，提问作者CncMess