如何用Scrapy和Playwright抓取事件触发的房产详情页链接

一、环境准备 #

先安装所需依赖：

• 安装Scrapy和Playwright：

  pip install scrapy playwright
  

• 安装Playwright的Chromium驱动：

  playwright install chromium
  

二、配置Scrapy项目 #

1. 创建项目及爬虫

   scrapy startproject magicbricks_scraper
   cd magicbricks_scraper
   scrapy genspider mumbai_properties magicbricks.com
   

2. 启用Playwright中间件
   在项目的settings.py中修改下载中间件配置，启用Playwright支持：

   DOWNLOADER_MIDDLEWARES = {
       'scrapy_playwright.middleware.PlaywrightMiddleware': 543,
   }
   
   # 配置Playwright浏览器参数
   PLAYWRIGHT_BROWSER_TYPE = 'chromium'
   PLAYWRIGHT_LAUNCH_OPTIONS = {
       'headless': False,  # 调试阶段可设为False，直观查看浏览器操作
       'args': ['--start-maximized'],
   }
   

三、编写爬虫逻辑 #

核心思路：利用Playwright模拟真实浏览器行为，等待页面动态加载完成后，优先提取卡片元素中隐藏的链接属性；若没有则模拟点击卡片，获取新打开页面的URL。

以下是完整爬虫代码（spiders/mumbai_properties.py）：

import scrapy
from scrapy_playwright.page import PageCoroutine

class MumbaiPropertiesSpider(scrapy.Spider):
    name = 'mumbai_properties'
    allowed_domains = ['magicbricks.com']
    start_urls = [
        'https://www.magicbricks.com/property-for-sale/residential-real-estate?bedroom=&proptype=Multistorey-Apartment,Builder-Floor-Apartment,Penthouse,Studio-Apartment&cityName=Mumbai'
    ]

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={
                    'playwright': True,
                    'playwright_page_coroutines': [
                        # 等待房产列表卡片加载完成，选择器需根据页面实际结构调整
                        PageCoroutine('wait_for_selector', '.mb-srp__card'),
                        # 滚动到底部触发懒加载（如果页面有）
                        PageCoroutine('evaluate', 'window.scrollTo(0, document.body.scrollHeight)'),
                        PageCoroutine('wait_for_timeout', 2000),  # 等待懒加载内容加载完成
                    ]
                }
            )

    async def parse(self, response):
        page = response.meta['playwright_page']
        
        # 获取所有房产卡片元素
        property_cards = await page.query_selector_all('.mb-srp__card')
        
        for card in property_cards:
            # 方式1：优先提取卡片隐藏的链接属性（效率更高）
            detail_link = await card.get_attribute('data-href')
            if detail_link:
                # 补全相对路径为完整URL
                if not detail_link.startswith('http'):
                    detail_link = response.urljoin(detail_link)
                yield {'detail_url': detail_link}
                continue
            
            # 方式2：无隐藏属性时，模拟点击卡片获取新页面URL
            with page.context.expect_page() as page_info:
                await card.click()
            new_page = await page_info.value
            detail_url = new_page.url
            yield {'detail_url': detail_url}
            # 及时关闭新页面，释放浏览器资源
            await new_page.close()
        
        # 处理分页：定位下一页按钮并发起请求
        next_btn = await page.query_selector('a[data-label="Next"]')
        if next_btn:
            next_url = await next_btn.get_attribute('href')
            if next_url:
                yield scrapy.Request(
                    response.urljoin(next_url),
                    meta={
                        'playwright': True,
                        'playwright_page_coroutines': [
                            PageCoroutine('wait_for_selector', '.mb-srp__card'),
                        ]
                    },
                    callback=self.parse
                )
        
        # 关闭当前页面
        await page.close()

四、关键注意事项 #

• 选择器适配：页面元素的选择器（如卡片、分页按钮）需根据网站实际HTML结构调整，建议用浏览器开发者工具定位元素。
• 反爬规避：在settings.py中设置自定义USER_AGENT，添加DOWNLOAD_DELAY = 2设置请求延迟，降低被封禁风险。
• 性能优化：优先用提取隐藏属性的方式获取链接，比模拟点击更高效；必须点击时，记得及时关闭新页面释放资源。

内容的提问来源于stack exchange，提问作者Aman Rao