如何用Selenium/Scrapy/Python下载网页内嵌PDF文件

方法一：使用Selenium获取并保存Blob PDF #

Selenium可模拟浏览器环境，通过执行JavaScript提取Blob的二进制内容，再转换为本地文件：

1. 安装依赖：

pip install selenium webdriver-manager

2. 示例代码：

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
import base64

# 初始化浏览器
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.get("https://www.rad.cvm.gov.br/ENET/frmExibirArquivoIPEExterno.aspx?NumeroProtocoloEntrega=1106753")

# 等待PDF加载完成（可根据页面实际情况调整等待时间或用显式等待）
driver.implicitly_wait(10)

# 执行JS获取Blob的base64编码内容（需根据页面实际元素调整选择器）
pdf_base64 = driver.execute_script("""
    const pdfElement = document.querySelector('embed[src^="blob:"]') || document.querySelector('iframe[src^="blob:"]');
    if (!pdfElement) return null;
    const xhr = new XMLHttpRequest();
    xhr.open('GET', pdfElement.src, false);
    xhr.responseType = 'blob';
    xhr.send();
    return new Promise((resolve) => {
        const reader = new FileReader();
        reader.onloadend = () => resolve(reader.result.split(',')[1]);
        reader.readAsDataURL(xhr.response);
    });
""")

# 保存为PDF文件
if pdf_base64:
    with open("downloaded_pdf.pdf", "wb") as f:
        f.write(base64.b64decode(pdf_base64))

driver.quit()

说明：如果页面有加载动画，建议用Selenium的WebDriverWait显式等待元素加载完成，避免因元素未就绪导致失败。

方法二：分析页面请求，直接调用API下载（更高效） #

Blob URL是前端生成的，背后通常对应后台接口返回的原始PDF文件。可通过浏览器开发者工具查找真实请求：

1. 打开目标页面，按F12打开开发者工具，切换到「网络」标签。
2. 刷新页面，筛选「文档」或「XHR」类型请求，找到返回PDF的请求（响应头含Content-Type: application/pdf）。
3. 复制该请求的URL、参数和请求头，用requests直接下载：

示例代码：

import requests

# 替换为找到的真实PDF接口URL
url = "这里替换为实际的PDF接口地址"
params = {
    "NumeroProtocoloEntrega": "1106753",
    # 其他必要参数从网络请求中复制
}

headers = {
    # 复制浏览器请求头中的Cookie、User-Agent等信息，避免被拦截
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    "Cookie": "这里替换为页面的Cookie值"
}

response = requests.get(url, params=params, headers=headers)
if response.status_code == 200 and response.headers.get("Content-Type") == "application/pdf":
    with open("direct_download.pdf", "wb") as f:
        f.write(response.content)

这种方法无需渲染页面，效率更高，但需手动分析请求参数和接口地址。

方法三：使用Scrapy结合Playwright渲染页面 #

若用Scrapy爬虫框架，可集成Playwright处理动态渲染的Blob内容：

1. 安装依赖：

pip install scrapy playwright
playwright install chrome

2. 示例Spider代码：

import scrapy
from scrapy_playwright.page import PageCoroutine
import base64

class PdfSpider(scrapy.Spider):
    name = "pdf_spider"
    start_urls = ["https://www.rad.cvm.gov.br/ENET/frmExibirArquivoIPEExterno.aspx?NumeroProtocoloEntrega=1106753"]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={
                    "playwright": True,
                    "playwright_page_coroutines": [
                        PageCoroutine("wait_for_selector", "embed[src^='blob:']"),
                    ],
                },
            )

    async def parse(self, response):
        page = response.meta["playwright_page"]
        # 执行JS获取Blob的base64内容
        pdf_base64 = await page.evaluate("""
            async () => {
                const pdfElement = document.querySelector('embed[src^="blob:"]');
                const xhr = new XMLHttpRequest();
                xhr.open('GET', pdfElement.src, false);
                xhr.responseType = 'blob';
                xhr.send();
                return new Promise((resolve) => {
                    const reader = new FileReader();
                    reader.onloadend = () => resolve(reader.result.split(',')[1]);
                    reader.readAsDataURL(xhr.response);
                });
            }
        """)
        if pdf_base64:
            with open("scrapy_pdf.pdf", "wb") as f:
                f.write(base64.b64decode(pdf_base64))
        await page.close()

需在Scrapy的settings.py中启用Playwright中间件：

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}
PLAYWRIGHT_LAUNCH_OPTIONS = {"headless": True}

内容的提问来源于stack exchange，提问作者Higo Felipe Silva Pires