如何用Python抓取第60届金马奖影片《石门》？异步下载遇阻求助

核心问题分析 #

你的异步请求全部失败，核心原因集中在这几点：

• 每次请求都新建ClientSession，频繁创建销毁连接会触发网站反爬机制，同时造成不必要的性能浪费
• 缺少必要请求头（如User-Agent、Referer），服务器会拒绝无标识的非浏览器请求
• 并发请求无限制，瞬间大量请求容易被拦截
• 异常处理不够细化，无法精准定位问题

针对性修改方案 #

1. 复用ClientSession：整个下载流程只创建一次会话，减少连接开销
2. 添加模拟请求头：模拟真实浏览器请求，绕过基础反爬校验
3. 控制并发数：用信号量限制同时发起的请求数量，避免触发反爬阈值
4. 优化异常与重试逻辑：细化异常类型，重试前添加延迟，提升成功率
5. 修正文件名处理：确保生成标准的.ts后缀文件，避免播放识别问题

修改后的完整代码 #

import asyncio
import aiohttp
import aiofiles
import os
import re


async def get_urls_from_m3u8(session, m3u8_url):
    async with session.get(m3u8_url) as response:
        if response.status == 200:
            content = await response.text()
            urls = re.findall(r'https?://[^\s]+\.jpeg', content)
            return urls
        else:
            print(f"获取M3U8文件失败，状态码: {response.status}")
            return []


async def download_image(session, url, directory, max_retries=3):
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
        'Referer': 'https://www.fofoyy.com/'
    }
    
    for attempt in range(max_retries + 1):
        try:
            async with session.get(url, headers=headers, timeout=60) as response:
                if response.status == 200:
                    filename_match = re.search(r'(\d+)\.jpeg', url)
                    if not filename_match:
                        print(f"无法解析URL文件名: {url}")
                        return False
                    ts_filename = f"{filename_match.group(1)}.ts"
                    ts_filepath = os.path.join(directory, ts_filename)

                    async with aiofiles.open(ts_filepath, 'wb') as ts_file:
                        while True:
                            chunk = await response.content.read(4096)
                            if not chunk:
                                break
                            await ts_file.write(chunk)

                    print(f"已保存: {ts_filename}")
                    return True
                elif response.status == 403:
                    print(f"请求被拒绝(403)，疑似反爬限制: {url}")
                    return False
                else:
                    print(f"下载失败，状态码: {response.status}，URL: {url}")
                    if attempt < max_retries:
                        await asyncio.sleep(1)
                    continue
        except asyncio.TimeoutError:
            print(f"请求超时，重试中 ({attempt+1}/{max_retries+1}): {url}")
            if attempt < max_retries:
                await asyncio.sleep(2)
        except Exception as e:
            if attempt < max_retries:
                print(f"请求出错，重试中 ({attempt+1}/{max_retries+1}): {str(e)}")
                await asyncio.sleep(1)
            else:
                print(f"最终失败，URL: {url}，错误: {str(e)}")
                return False
    return False


async def main():
    directory = "downloaded_files_ts"
    if not os.path.exists(directory):
        os.makedirs(directory)

    m3u8_url = 'https://v8.longshengtea.com/yyv8/202310/06/2yJDc3LMsW1/video/2000k_0X1080_64k_25/hls/index.m3u8'
    
    async with aiohttp.ClientSession() as session:
        urls = await get_urls_from_m3u8(session, m3u8_url)
        if not urls:
            print("未提取到任何分片URL")
            return
        
        semaphore = asyncio.Semaphore(10)
        
        async def bounded_download(url):
            async with semaphore:
                return await download_image(session, url, directory)
        
        tasks = [bounded_download(url) for url in urls]
        await asyncio.gather(*tasks)
    
    print("所有下载任务完成")


if __name__ == '__main__':
    asyncio.run(main())

额外说明 #

• 信号量Semaphore(10)限制同时发起10个请求，可根据实际情况调整数值
• 增大了读取块大小至4KB，提升下载效率
• 针对403状态码单独提示反爬问题，方便排查
• 重试前添加延迟，降低被服务器识别为恶意请求的概率

内容的提问来源于stack exchange，提问作者Ethan Lee