Python批量下载指定编号网站PNG文件的异常问题排查

尝试下载某网站目录下编号1到311的.png文件，已创建对应编号的数组，打算通过格式化URL替换编号，但程序未按预期下载指定文件，反而尝试下载URL的每个字符，导致下载失败。相关代码如下：

import requests
from multiprocessing.pool import ThreadPool
 
def download_url(url):
  print("downloading: ",url.format(*skins))
  # assumes that the last segment after the / represents the file name
  # if url is abc/xyz/file.txt, the file name will be file.txt
  file_name_start_pos = url.rfind("/") + 1
  file_name = url[file_name_start_pos:]
 
  r = requests.get(url.format(*skins), stream=True)
  if r.status_code == requests.codes.ok:
    with open(file_name, 'wb') as f:
      for data in r:
        f.write(data)
  return url.format(*skins)
 
skins = ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39', '40', '41', '42', '43', '44', '45', '46', '47', '48', '49', '50', '51', '52', '53', '54', '55', '56', '57', '58', '59', '60', '61', '62', '63', '64', '65', '66', '67', '68', '69', '70', '71', '72', '73', '74', '75', '76', '77', '78', '79', '80', '81', '82', '83', '84', '85', '86', '87', '88', '89', '90', '91', '92', '93', '94', '95', '96', '97', '98', '99', '100', '101', '102', '103', '104', '105', '106', '107', '108', '109', '110', '111', '112', '113', '114', '115', '116', '117', '118', '119', '120', '121', '122', '123', '124', '125', '126', '127', '128', '129', '130', '131', '132', '133', '134', '135', '136', '137', '138', '139', '140', '141', '142', '143', '144', '145', '146', '147', '148', '149', '150', '151', '152', '153', '154', '155', '156', '157', '158', '159', '160', '161', '162', '163', '164', '165', '166', '167', '168', '169', '170', '171', '172', '173', '174', '175', '176', '177', '178', '179', '180', '181', '182', '183', '184', '185', '186', '187', '188', '189', '190', '191', '192', '193', '194', '195', '196', '197', '198', '199', '200', '201', '202', '203', '204', '205', '206', '207', '208', '209', '210', '211', '212', '213', '214', '215', '216', '217', '218', '219', '220', '221', '222', '223', '224', '225', '226', '227', '228', '229', '230', '231', '232', '233', '234', '235', '236', '237', '238', '239', '240',  '241', '242', '243', '244', '245', '246', '247', '248', '249', '250',  '251', '252', '253', '254', '255', '256', '257', '258', '259', '260',  '261', '262', '263', '264', '265', '266', '267', '268', '269', '270',  '271', '272', '273', '274', '275', '276', '277', '278', '279', '280',  '281', '282', '283', '284', '285', '286', '287', '288', '289', '290',  '291', '292', '293', '294', '295', '296', '297', '298', '299', '300',  '301', '302', '303', '304', '305', '306', '307', '308', '309', '310',  '311'] 
url = 'https://test-site.com/skins/{0}.png'
#urls = ["https://test-site.com/skins/*.png"]
 
# Run 5 multiple threads. Each call will take the next element in urls list
for x in skins:
    results = ThreadPool(5).imap_unordered(download_url, url.format(*skins))
    for r in results:
        print(r)

错误原因 #

1. URL格式化逻辑错误：url.format(*skins)会把数组里所有编号一次性传入只有一个占位符的URL模板，生成类似https://test-site.com/skins/1.png2.png3.png...311.png的畸形URL。
2. 线程池参数传递错误：把畸形的长URL传给imap_unordered，而字符串在Python里是可迭代对象，线程池会把URL的每个字符当作单独任务去下载，这就是出现“下载每个字符”的原因。
3. 下载函数冗余错误：函数内部重复调用url.format(*skins)，进一步放大URL错误，同时文件名提取也是基于错误的输入URL。

修正后的代码 #

import requests
from multiprocessing.pool import ThreadPool

def download_url(skin_id):
    # 针对每个编号单独生成正确的URL
    url = f'https://test-site.com/skins/{skin_id}.png'
    print("downloading: ", url)
    # 直接用编号作为文件名，避免URL解析错误
    file_name = f'{skin_id}.png'
    
    try:
        r = requests.get(url, stream=True)
        r.raise_for_status()  # 捕获HTTP请求错误
        # 分块写入文件，提升大文件下载效率
        with open(file_name, 'wb') as f:
            for chunk in r.iter_content(chunk_size=1024):
                f.write(chunk)
        return f"成功下载: {url}"
    except Exception as e:
        return f"下载失败 {url}: {str(e)}"

# 自动生成1-311的编号字符串，不用手动写数组
skins = [str(i) for i in range(1, 312)]

# 只创建一次线程池，避免重复初始化开销
with ThreadPool(5) as pool:
    # 把所有编号作为任务传入线程池
    results = pool.imap_unordered(download_url, skins)
    for result in results:
        print(result)

修正说明 #

• 简化URL生成：每个编号单独拼接正确的URL，彻底避免批量格式化的错误。
• 优化线程池使用：仅创建一次线程池，直接传入编号数组作为任务，避免字符串迭代导致的错误。
• 增加错误处理：捕获HTTP请求异常，返回清晰的成功/失败提示。
• 简化数组生成：用列表推导式自动生成编号，减少手动输入的繁琐和出错概率。

内容的提问来源于stack exchange，提问作者Michael Griffith