Python+Curl实现从ASP网站下载PDF并按网页显示名命名

需求说明 #

需要从ASP网站批量下载PDF文件，并按照网页上<a>标签的显示文本作为文件名保存，例如：

Chapter 3 - Weird science
将该PDF保存为Chapter 3 - Weird science.pdf

修改后的代码 #

from bs4 import BeautifulSoup
import requests
from urllib import parse as urlparse
import os

# 配置要爬取的班级和科目
klassen = ['1e klas']
vakken = ['Wiskunde']
# 可选科目列表
'''['Engels','Aardrijkskunde','Economie', 'Filosofie','Frans', 'Geschiedenis', 
          'Nask', 'Natuurkunde', 'Nederlands', 'Scheikunde', 'Spaans', 'Wiskunde',
          'Biologie', 'Duits', 'Grieks','Latijn','Leesmateriaal', 
          'Loopbaanorientatie','NLT']'''

# 遍历班级和科目
for klas in klassen: 
    for vak in vakken: 
        url = "https://www.svpo.nl/curriculum.asp"
        payload = f'vak={vak}&klas_en_schoolsoort={klas}'
        headers = {
            'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
            'Content-Type': 'application/x-www-form-urlencoded',
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
        }

        # 发送POST请求获取页面内容
        response = requests.post(url, data=payload, headers=headers)
        response.raise_for_status()  # 捕获请求错误

        # 创建保存目录
        path_out = r'c:\books\\'
        path = os.path.join(path_out, klas, vak)
        os.makedirs(path, exist_ok=True)  # 简化目录创建

        # 解析HTML页面
        soup = BeautifulSoup(response.text, "lxml")
        pdf_links = soup.find_all('a', href=True)

        # 遍历所有PDF链接
        for link in pdf_links:
            current_link = link['href']
            if current_link.endswith('pdf'):
                # 获取网页显示的文本作为文件名
                raw_filename = link.get_text(strip=True)
                # 替换Windows文件名中的非法字符
                invalid_chars = '<>:"/\\|?*'
                for char in invalid_chars:
                    raw_filename = raw_filename.replace(char, '_')
                # 添加PDF后缀
                filename = f"{raw_filename}.pdf"
                file_path = os.path.join(path, filename)

                # 处理链接中的转义字符，转换为合法URL
                fixed_link = current_link.replace('\\', '/')
                # 对文件名部分进行URL编码
                base_url, file_part = fixed_link.rsplit('/', 1)
                encoded_file = urlparse.quote(file_part)
                fixed_link = f"{base_url}/{encoded_file}"

                # 使用requests直接下载文件，替代curl调用
                try:
                    pdf_response = requests.get(fixed_link, headers=headers, stream=True)
                    pdf_response.raise_for_status()
                    with open(file_path, 'wb') as f:
                        for chunk in pdf_response.iter_content(chunk_size=8192):
                            f.write(chunk)
                    print(f"已保存: {file_path}")
                except Exception as e:
                    print(f"下载失败 {fixed_link}: {str(e)}")

关键修改点 #

• 清理冗余导入：移除重复的urllib.request as requests，保留标准requests库
• 修复缩进错误：将请求代码放回科目循环内，确保每个科目都能发起独立请求
• 自定义文件名：用link.get_text(strip=True)提取网页显示文本，处理后作为文件名
• 兼容Windows文件系统：替换文件名中的非法特殊字符，避免保存失败
• 替换系统调用：改用requests直接下载文件，无需依赖系统curl命令，稳定性更强
• 优化目录处理：用os.path.join和exist_ok=True简化目录创建逻辑
• 增加错误捕获：添加请求和下载环节的异常处理，便于排查问题

内容的提问来源于stack exchange，提问作者user9118870