Scrapy爬取分页URL不变的站点多页数据实现方案

目标站点是ASP.NET WebForm架构，分页切换不会修改URL，通过提交页面内置的隐藏表单字段（__VIEWSTATE、__VIEWSTATEGENERATOR、__EVENTVALIDATION）发POST请求实现分页，直接GET固定URL只能拿到第一页数据。

• 首次GET请求拿到第一页内容后，先提取当前页所有律师详情链接，交给详情解析函数处理
• 从第一页的分页栏提取总页码，确定翻页终止条件
• 每一页处理完后，提取当前页所有隐藏表单字段，补充分页触发参数，构造POST请求请求下一页
• 逐页递推直到到达最后一页，即可拿到全量数据
• 原配置的单域名并发1、1秒请求延迟刚好适配该场景，ASP.NET的隐藏参数和页序强绑定，乱序并发会导致参数失效

import scrapy
from scrapy.http import Request, FormRequest


class TestSpider(scrapy.Spider):
    name = 'test'
    start_urls = ['https://www.ifep.ro/justice/lawyers/lawyerspanel.aspx']
    custom_settings = {
        'CONCURRENT_REQUESTS_PER_DOMAIN': 1,
        'DOWNLOAD_DELAY': 1,
        'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36'
    }

    def parse(self, response, current_page=1, total_page=None):
        # 提取当前页所有律师详情页链接
        lawyer_links = response.xpath("//div[@class='list-group']//@href").extract()
        for link in lawyer_links:
            url = response.urljoin(link)
            if url.endswith('.ro') or url.endswith('.ro/'):
                continue
            yield Request(url, callback=self.parse_book)

        # 第一页加载时计算总页数
        if total_page is None:
            page_nums = []
            for page_text in response.xpath("//div[contains(@class,'pager')]//a/text()").getall():
                page_text = page_text.strip()
                if page_text.isdigit():
                    page_nums.append(int(page_text))
            total_page = max(page_nums) if page_nums else 1

        # 构造下一页请求
        if current_page < total_page:
            next_page = current_page + 1
            # 收集当前页所有隐藏表单参数
            formdata = {}
            for hidden in response.xpath("//input[@type='hidden']"):
                name = hidden.xpath("./@name").get()
                value = hidden.xpath("./@value").get("")
                if name:
                    formdata[name] = value
            # 补充分页触发参数
            formdata["__EVENTTARGET"] = "AspNetPager1"
            formdata["__EVENTARGUMENT"] = str(next_page)
            
            yield FormRequest(
                url=response.url,
                formdata=formdata,
                callback=self.parse,
                cb_kwargs={"current_page": next_page, "total_page": total_page}
            )

    def parse_book(self, response):
        title = response.xpath("//span[@id='HeadingContent_lblTitle']//text()").get()
        d1 = response.xpath("//div[@class='col-md-10']//p[1]//text()").get()
        d1 = d1.strip() if d1 else ""
        d2 = response.xpath("//div[@class='col-md-10']//p[2]//text()").get()
        d2 = d2.strip() if d2 else ""
        d3 = response.xpath("//div[@class='col-md-10']//p[3]//span//text()").get()
        d3 = d3.strip() if d3 else ""
        d4 = response.xpath("//div[@class='col-md-10']//p[4]//text()").get()
        d4 = d4.strip() if d4 else ""

        yield {
            "title1": title,
            "title2": d1,
            "title3": d2,
            "title4": d3,
            "title5": d4,
        }

• 如果分页请求返回错误，打开浏览器F12网络面板，手动点击一次分页，查看POST请求体里的__EVENTTARGET实际值，替换代码中对应字符串即可
• 代码新增了字段非空判断，避免个别律师详情页字段缺失导致爬虫中断
• 不要随意调高并发数、调低延迟，否则大概率触发站点反爬或者参数校验失败

内容的提问来源于stack exchange，提问作者Amen Aziz