如何在Scrapy中实现多页并发分页爬取与数据解析

问题修复与并发爬取实现方案 #

1. 现有分页不生效的根因 #

你将分页处理代码放在了处理房源详情页的parse_item函数中，详情页不存在列表页才有的分页导航元素，永远无法获取到下一页链接，所以分页功能失效。只需将分页逻辑移动到处理列表页的parse函数末尾即可。

2. 并发爬取实现方法 #

Scrapy 本身是异步架构，天生支持并发请求，你之前认为start_requests中循环生成页面请求是逐页执行是误解：只要一次性把所有90页的请求提交到请求队列，Scrapy会按照配置的并发数同时发起请求，无需等待上一页爬取完成，完全满足快速爬取的需求。

3. 优化后完整代码 #

# -*- coding: utf-8 -*-
import scrapy
from scrapy import Request
from datetime import datetime
from scrapy.crawler import CrawlerProcess

dt_today = datetime.now().strftime('%Y%m%d')
file_name = dt_today+' HPI Data'


# Create Spider class
class UneguiApartments(scrapy.Spider):
    name = "unegui_apts"
    allowed_domains = ["www.unegui.mn"]
    custom_settings = {
        "FEEDS": {f'{file_name}.csv': {'format': 'csv'}},
        # 并发请求数，可根据实际情况调整，不要太高避免被网站封禁
        "CONCURRENT_REQUESTS": 16,
        # 下载延迟，单位秒，降低被封风险
        "DOWNLOAD_DELAY": 0.5,
        # 关闭重试可以加快速度，需要稳定性可以打开
        "RETRY_ENABLED": False
    }

    def start_requests(self):
        # 方案1：批量生成所有90页请求，直接并发爬取，速度最快
        base_url = "https://www.unegui.mn/l-hdlh/l-hdlh-zarna/oron-suuts-zarna/ulan-bator/?cities=1&page={}"
        for page in range(1, 91):
            yield Request(base_url.format(page), self.parse)
        
        # 方案2：如果不想批量生成，用原来的逐页跟进方式，替换掉上面的循环即可
        # urls = ['https://www.unegui.mn/l-hdlh/l-hdlh-zarna/oron-suuts-zarna/ulan-bator/']
        # for url in urls:
        #     yield Request(url, self.parse)

    def parse(self, response, **kwargs):
        cards = response.xpath("//li[contains(@class,'announcement-container')]")

        # parse details
        for card in cards:
            name = card.xpath(".//a[@itemprop='name']/@content").extract_first()
            price = card.xpath(".//*[@itemprop='price']/@content").extract_first()
            rooms = card.xpath(".//div[contains(@class,'announcement-block__breadcrumbs')]/text()").extract_first().split('»')[0].strip()
            link = card.xpath(".//a[@itemprop='url']/@href").extract_first()
            date_block = card.xpath("normalize-space(.//div[contains(@class,'announcement-block__date')]/text())").extract_first().split(',')
            date = date_block[0].strip()
            city = date_block[1].strip()

            item = {'name': name,
                    'date': date,
                    'rooms': rooms,
                    'price': price,
                    'city': city,
                    }
            # follow absolute link to scrape deeper level
            yield response.follow(link, callback=self.parse_item, meta={'item': item})

        # 分页逻辑移到这里，只在列表页执行，如果用方案1批量生成请求可以注释掉下面这段
        # next_page = response.xpath("//a[contains(@class,'red')]/parent::li/following-sibling::li/a/@href").extract_first()
        # if next_page:
        #     yield response.follow(next_page, callback=self.parse)

    def parse_item(self, response):
        # retrieve previously scraped item between callbacks
        item = response.meta['item']

        # parse additional details
        list_span = response.xpath(".//span[contains(@class,'value-chars')]//text()").extract()
        list_a = response.xpath(".//a[contains(@class, 'value-chars')]//text()").extract()

        # get additional details from list of <span> tags, element by element
        floor_type = list_span[0].strip()
        num_balcony = list_span[1].strip()
        garage = list_span[2].strip()
        window_type = list_span[3].strip()
        door_type = list_span[4].strip()
        num_window = list_span[5].strip()

        # get additional details from list of <a> tags, element by element
        commission_year = list_a[0].strip()
        num_floors = list_a[1].strip()
        area_sqm = list_a[2].strip()
        floor = list_a[3].strip()
        leasing = list_a[4].strip()
        district = list_a[5].strip()
        address = list_a[6].strip()

        # update item with newly parsed data
        item.update({
            'district': district,
            'address': address,
            'area_sqm': area_sqm,
            'floor': floor,
            'commission_year': commission_year,
            'num_floors': num_floors,
            'num_windows': num_window,
            'num_balcony': num_balcony,
            'floor_type': floor_type,
            'window_type': window_type,
            'door_type': door_type,
            'garage': garage,
            'leasing': leasing
        })
        yield item


# main driver
if __name__ == "__main__":
    process = CrawlerProcess()
    process.crawl(UneguiApartments)
    process.start()

4. 补充说明 #

• CSV西里尔字符显示问题你已经找到的Excel导入方案是通用解决方案，无需额外调整代码。
• 并发数CONCURRENT_REQUESTS可根据你的网络情况和网站反爬策略调整，最高建议不要超过32，避免触发网站封禁规则。
• 如果网站有反爬限制，可以开启RETRY_ENABLED，适当调高DOWNLOAD_DELAY。

内容的提问来源于stack exchange，提问作者WX1505