如何使用Beautiful Soup跳转多页面并下载名称含Public Comment的PDF

调整后可实现需求的完整代码 #

import requests
from bs4 import BeautifulSoup

base_url = "https://www.ci.atherton.ca.us/"
start_url = "https://www.ci.atherton.ca.us/Archive.aspx?AMID=41"
adid_key = "Archive.aspx?ADID="
pdf_keyword = "Public Comment"

# 先收集所有ADID链接
adid_links = []
soup = BeautifulSoup(requests.get(start_url).content, "html.parser")
for link in soup.find_all("a"):
    href = link.get("href", "")
    if adid_key in href:
        full_adid_url = base_url + href.lstrip("/")
        adid_links.append(full_adid_url)

# 遍历每个ADID页面找符合要求的PDF
for page_url in adid_links:
    print(f"正在处理页面：{page_url}")
    try:
        page_soup = BeautifulSoup(requests.get(page_url, timeout=10).content, "html.parser")
        for pdf_link in page_soup.find_all("a"):
            pdf_href = pdf_link.get("href", "")
            # 筛选PDF链接
            if not pdf_href.lower().endswith(".pdf"):
                continue
            # 匹配关键词：链接文本 或者 PDF文件名包含关键词
            link_text = pdf_link.get_text(strip=True)
            pdf_filename = pdf_href.split("/")[-1]
            if pdf_keyword in link_text or pdf_keyword in pdf_filename:
                full_pdf_url = base_url + pdf_href.lstrip("/")
                # 下载PDF
                print(f"正在下载：{full_pdf_url}")
                pdf_content = requests.get(full_pdf_url, timeout=10).content
                with open(pdf_filename, "wb") as f:
                    f.write(pdf_content)
                print(f"{pdf_filename} 下载完成")
    except Exception as e:
        print(f"处理页面{page_url}出错：{str(e)}，跳过该页面")

说明 #

• 代码会先收集所有带Archive.aspx?ADID=的链接，再逐个访问解析页面内容
• 自动识别页面内所有PDF链接，只要链接显示文本、或者PDF文件名包含"Public Comment"就会自动下载
• 下载的PDF默认保存在脚本运行的当前目录下，文件名和站点原文件名保持一致
• 增加了超时和异常捕获逻辑，单个页面请求失败不会中断整体运行
• 如果请求出现403错误，可以在requests.get方法中加入headers参数模拟浏览器请求即可，示例：

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
}
# 调用时传入headers
requests.get(page_url, headers=headers, timeout=10)

内容的提问来源于stack exchange，提问作者lsignori