Scrapy实现多页数据爬取并设置翻页上限的方法

• 下一页链接提取逻辑错误：原代码用商品循环结束后的card节点提取分页链接，分页按钮属于页面全局元素，不属于商品子节点，永远无法匹配到有效地址，导致翻页逻辑直接终止，只能爬取第一页。
• 缺少页码计数逻辑，无法自定义最大爬取页数。

import scrapy
from scrapy.item import Field
from itemloaders.processors import TakeFirst
from scrapy.crawler import CrawlerProcess


class StatisticsItem(scrapy.Item):
    ebay_div = Field(output_processor=TakeFirst())
    url = Field(output_processor=TakeFirst())


class StatisticsSpider(scrapy.Spider):
    name = 'ebay'
    max_pages = 5  # 自定义最大爬取页数，可按需修改
    page_count = 1 # 记录当前已爬取的页码
    start_urls = ['https://www.ebay.com/b/Collectible-Card-Games-Accessories/2536/bn_1852210?rt=nc&LH_BIN=1' +
                  '&LH_PrefLoc=2&mag=1&_sop=16']

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url)

    def parse(self, response):
        all_cards = response.xpath('//div[@class="s-item__wrapper clearfix"]')
        for card in all_cards:
            name = card.xpath('.//h3/text()').get() # 获取商品名称
            price = card.xpath('.//span[@class="s-item__price"]//text()').get() # 获取商品价格
            product_url = card.xpath('.//a[@class="s-item__link"]//@href').get() # 获取商品链接

            summary_data = {
                "Name": name,
                "Price": price,
                "URL": product_url
            }
            data = {'summary_data': summary_data}
            yield scrapy.Request(product_url, meta=data, callback=self.parse_product_details)

        # 从页面全局提取下一页链接
        next_page_url = response.xpath('//a[@class="pagination__next icon-link"]/@href').extract_first()
        # 校验下一页链接有效性、是否达到最大爬取页数
        if next_page_url is None or str(next_page_url).endswith("#") or self.page_count >= self.max_pages:
            self.log("eBay products collected successfully !!!")
        else:
            self.page_count += 1
            print('\n' + '-' * 30)
            print(f'Next page: {next_page_url}, 当前为第{self.page_count}页')
            yield scrapy.Request(next_page_url, callback=self.parse)

    def parse_product_details(self, response):
        data = response.meta['summary_data']
        data['location'] = response.xpath('//span[@itemprop="availableAtOrFrom"]/text()').extract_first()
        yield data


process = CrawlerProcess(
    settings={
        'FEED_URI': 'collectible_cards.json',
        'FEED_FORMAT': 'jsonlines'
    }
)
process.crawl(StatisticsSpider)
process.start()

• 新增类属性max_pages，可自定义最大爬取页数，默认值为5，按需调整即可
• 新增类属性page_count，用于记录当前已爬取的页码数
• 修正下一页链接的提取逻辑，改为从页面全局response对象查询，正确匹配分页按钮
• 翻页前新增页码校验，达到最大页数后自动终止翻页，避免无限制爬取

内容的提问来源于stack exchange，提问作者me.limes