Scrapy搭配Selenium实现多URL循环爬取失效的问题求解

DevOps 小满

2026-11-7

原代码失效的核心问题

逻辑错位：把全量URL的Selenium加载循环放在__init__初始化阶段，循环跑完后self.html只会存储最后一个URL的页面源码。当Scrapy按自身调度逻辑依次处理每个start_url进入parse方法时，读取的self.html和当前处理的URL完全不对应，自然无法抓取正确数据
配置错误：allowed_domains配置为trc.com，和实际爬取的google.com不匹配，请求会被Scrapy内置的域名过滤机制拦截
流程无容错：隐私弹窗的点击逻辑没有做异常捕获，当后续URL加载时不弹出授权按钮，find_element_by_xpath会直接抛出元素不存在的错误，中断整个循环
违背Scrapy设计逻辑：将Selenium作为独立的批量加载脚本运行，没有和Scrapy的单请求生命周期绑定，和Scrapy的异步调度机制完全脱节

正确的实现思路

单请求绑定单页Selenium渲染：在__init__中只完成Selenium driver的初始化，不在初始化阶段加载任何URL。每处理一个Scrapy请求，就在parse方法中用driver加载当前请求的URL，完成弹窗处理、页面等待后提取数据，实现和Scrapy调度逻辑的对齐
修正域名配置，将allowed_domains调整为和爬取目标匹配的google.com
为弹窗点击、元素查找逻辑增加异常捕获，找不到弹窗元素时直接跳过点击步骤，不中断爬取流程
替换固定time.sleep为Selenium显式等待，等目标内容节点加载完成后再提取页面源码，避免因页面加载慢导致的内容缺失
增加爬虫退出钩子，自动关闭Selenium实例释放系统资源

参考修正代码

import scrapy
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from shutil import which
from scrapy.selector import Selector

class TrSpider(scrapy.Spider):
    name = 'tr'
    allowed_domains = ['google.com']
    start_urls = [
        # 保留你原有的所有目标URL即可
        'https://www.google.com/travel/things-to-do?g2lb=2502548,2503806,4258168,4270442,4306835,4317915,4328159,4371334,4401769,4419364,4463263,4463666,4464463,4482194,4482438,4486153,4270859,4284970,4291517&hl=en-PK&gl=pk&un=1&dest_mid=/m/0rtv&dest_state_type=main&sa=X&ved=2ahUKEwjvk6PXqLXuAhWN2BQKHZs1BNsQhdIBegQIARAW',
        'https://www.google.com/travel/things-to-do?g2lb=2502548,2503806,4258168,4270442,4306835,4317915,4328159,4371334,4401769,4419364,4463263,4463666,4464463,4482194,4482438,4486153,4270859,4284970,4291517&hl=en-PK&gl=pk&un=1&dest_mid=/m/012gg6&dest_state_type=main&sa=X&ved=2ahUKEwjvk6PXqLXuAhWN2BQKHZs1BNsQhdIBegQIARAc',
        # 其余原始start_urls按原样补充即可
    ]

    def __init__(self):
        super().__init__()
        chrome_option = Options()
        chrome_option.add_argument('--headless=new')
        chrome_option.add_argument('--disable-blink-features=AutomationControlled')
        # 仅初始化driver实例，不在此处加载任何URL
        service = Service(executable_path=which('chromedriver'))
        self.driver = webdriver.Chrome(service=service, options=chrome_option)
        # 设置最长10秒的显式等待，替代固定sleep
        self.wait = WebDriverWait(self.driver, 10)

    def parse(self, response):
        # 将driver加载动作和当前Scrapy调度的URL绑定
        target_url = response.url
        self.driver.get(target_url)
        
        # 隐私弹窗处理逻辑，增加异常兼容
        try:
            consent_btn = self.wait.until(
                EC.element_to_be_clickable((By.XPATH, '//button[@class="VfPpkd-LgbsSe VfPpkd-LgbsSe-OWXEXe-INsAgc VfPpkd-LgbsSe-OWXEXe-dgl2Hf Rj2Mlf OLiIxf PDpWxe J3Eqid"]'))
            )
            consent_btn.click()
        except:
            # 无弹窗时直接跳过，不中断流程
            pass
        
        # 等核心内容节点加载完成后再提取源码
        self.wait.until(EC.presence_of_element_located((By.XPATH, '//div[@class="GwjAi "]')))
        page_source = self.driver.page_source
        resp = Selector(text=page_source)
        
        for item in resp.xpath('//div[@class="GwjAi "]'):
            yield {
                'Name': item.xpath('./div[1]/div/text()').get(),
                'Rating': item.xpath('./div[2]/span/span/span[1]/text()').get(),
                'Number of reviews': item.xpath('./div[2]/span/span/span[2]/text()').get(),
                'Description': item.xpath('./div[3]/text()').extract(),
                'Url': target_url
            }

    def closed(self, reason):
        # 爬虫退出时自动关闭driver，释放系统资源
        self.driver.quit()

内容的提问来源于stack exchange，提问作者Noob Gamer