如何用Python加速网页爬取？4308页医疗站点爬取提速需求

针对你爬取4308页耗时过长的问题，这几个方法能有效缩短爬取时间：

1. 改用并发请求（核心提速手段） #

原来的串行请求是逐页等待响应，效率极低。用异步请求库aiohttp可以同时发起多个请求，大幅提升爬取速度。示例代码如下：

import pandas as pd
import aiohttp
import asyncio
from bs4 import BeautifulSoup

async def fetch_page(session, page):
    url = f"https://annumed.sante-dz.com/annuaire?page={page}"
    # 模拟浏览器请求头，避免被识别为爬虫
    headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"}
    async with session.get(url, headers=headers, timeout=10) as response:
        content = await response.text()
        soup = BeautifulSoup(content, "lxml")  # 用lxml解析更快
        page_data = []
        for i in soup.select(".info:has(header)"):
            name = i.h3.text
            spec = i.select_one("header p").text
            cats = [s.text for s in i.select(".speciliate")]
            addr = i.select_one(".address__wrapper").get_text(strip=True, separator=" ")
            page_data.append((name, spec, ",".join(cats), addr))
        return page_data

async def main():
    data = []
    concurrency = 15  # 控制并发数，建议10-20，避免触发反爬
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_page(session, page) for page in range(1, 4308)]
        # 分批执行任务，避免一次性创建过多任务导致内存压力
        for i in range(0, len(tasks), concurrency):
            batch_tasks = tasks[i:i+concurrency]
            batch_results = await asyncio.gather(*batch_tasks)
            for result in batch_results:
                data.extend(result)
            print(f"已处理 {min(i+concurrency, 4307)} 页...")

    df = pd.DataFrame(data, columns=["Name", "Spec", "Categories", "Address"])
    excel_file_path = r"C:\Users\LENOVO\Desktop\scraping_sante\sectr_med.xlsx"
    df.to_excel(excel_file_path, index=False)
    print("数据已保存到Excel:", excel_file_path)

if __name__ == "__main__":
    asyncio.run(main())

2. 替换更快的HTML解析器 #

把BeautifulSoup默认的html.parser换成lxml，解析速度能提升数倍。先安装依赖：pip install lxml，再修改代码中的解析器参数：

soup = BeautifulSoup(requests.get(url).content, "lxml")

3. 添加请求头与超时设置 #

给请求加上浏览器UA，避免被服务器限制速度；同时设置超时时间，防止单个请求卡住拖慢整体进度：

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"}
response = requests.get(url, headers=headers, timeout=10)

4. 分批保存数据 #

如果爬取中途中断，之前的数据会丢失。可以每爬取几百页就保存一次进度：

# 在串行爬取的循环中添加：
if page % 500 == 0:
    temp_df = pd.DataFrame(data, columns=["Name", "Spec", "Categories", "Address"])
    temp_df.to_excel(excel_file_path, index=False)
    print(f"已保存到第 {page} 页数据")

5. 规避反爬限制 #

如果服务器有频率限制，可在并发请求中加入轻微随机延迟，避免被封同时保证速度：

# 在fetch_page函数中，请求前加入延迟
import random
await asyncio.sleep(random.uniform(0.1, 0.5))

内容的提问来源于stack exchange，提问作者Wiam.07 Lazazi