Scrapy多页爬取数据被覆盖仅获最后一页数据的解决方案咨询

问题根源 #

你在类定义阶段直接用for循环给payload和headers赋值，循环执行5次后，self.payload最终只会保留最后一次循环（x=4）的结果，导致start_requests仅发送最后一页的请求，自然只能获取最后一页数据。

修复方案 #

将多页请求的生成逻辑移到start_requests方法内部，动态生成每一页的请求，确保每一页的请求都被单独发送：

import scrapy
from scrapy import FormRequest
from scrapy.http import Request

class TestSpider(scrapy.Spider):
    name = 'test'
    url = 'https://advpalata.vrn.ru/registers/reestr_lawyers/'
    
    # 把headers定义为类属性，无需每次循环重复生成
    headers = {
        'authority': 'advpalata.vrn.ru',
        'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
        'accept-language': 'en-US,en;q=0.9',
        'cache-control': 'max-age=0',
        'content-type': 'application/x-www-form-urlencoded',
        'cookie': 'PHPSESSID=546743b283bb9e3b2e78dabbfb894220; ie=yes; stat_id=546743b283bb9e3b2e78dabbfb894220; _ym_uid=1658939896610936176; _ym_d=1658939896; _ym_isad=2; PHPSESSID=546743b283bb9e3b2e78dabbfb894220',
        'origin': 'https://advpalata.vrn.ru',
        'referer': 'https://advpalata.vrn.ru/registers/reestr_lawyers/',
        'sec-ch-ua': '"Not/A)Brand";v="99", "Google Chrome";v="103", "Chromium";v="103"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': '"Windows"',
        'sec-fetch-dest': 'document',
        'sec-fetch-mode': 'navigate',
        'sec-fetch-site': 'same-origin',
        'sec-fetch-user': '?1',
        'upgrade-insecure-requests': '1',
        'user-agent': 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
    }

    def start_requests(self):
        # 循环生成每一页的请求
        for x in range(0, 5):
            # 注意把&改成&，这是HTML转义字符，实际POST需要正常的分隔符
            payload = f'p={x}&letterfilter=%D0%90'
            yield scrapy.FormRequest(
                url=self.url,
                method='POST',
                body=payload,
                headers=self.headers,
                callback=self.parse_item,
            )

    def parse_item(self, response):
        books = response.xpath("//td[@class='name']//a//@href").extract()
        for book in books:
            absolute_url = response.urljoin(book)
            yield Request(absolute_url, callback=self.parse_book)
            
    def parse_book(self, response):
        title = response.css("h3::text").get()
        # phone = response.xpath("//div[@class='advocate-right']//p[@umi:field-name='phone']//text()").get()
        # email = response.xpath("//div[@class='advocate-right']//p[@umi:field-name='email']//a//text()").get()
        
        yield {
            'title': title,
            # 'phone': phone,
            # 'email': email
        }

额外说明 #

1. 将headers定义为类属性，避免每次循环重复生成，提升效率。
2. 修正了payload中的&为&，确保服务器能正确解析请求参数。
3. 如果需要爬取更多页数，只需调整range(0,5)的结束值即可（比如range(0,10)对应前10页）。

内容的提问来源于stack exchange，提问作者Amen Aziz