Scrapy Spider配置多URL失效，单个URL可正常运行求助

问题描述 #

编写的集成Selenium的Scrapy爬虫，用于爬取devgrossonline.com商品数据。单个分类URL配置时可正常运行，但多个分类URL配置时无法正常工作，输出结果不一致。

核心原因 #

Scrapy是异步IO框架，多个请求会并发执行。原代码中全局共用一个Selenium浏览器实例，当第二个请求调用self.driver.get()跳转新URL时，第一个请求正在执行的翻页、数据抓取操作会被中断，导致浏览器上下文混乱，最终数据抓取异常。

修复后的完整代码 #

import time
from datetime import datetime
from threading import Lock

import scrapy
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager


class DevgrossSpider(scrapy.Spider):
    name = 'devgross'
    allowed_domains = ['devgrossonline.com']
    start_urls = [
        'https://devgrossonline.com/sut-kahvaltilik',
        'https://devgrossonline.com/meyve-sebze',
        'https://devgrossonline.com/et-sarkuteri'
    ]

    def __init__(self, *args, **kwargs):
        super(DevgrossSpider, self).__init__(*args, **kwargs)
        self.driver_lock = Lock()
        # 初始化单个driver，通过锁保证同一时间仅一个请求使用
        self.driver = webdriver.Chrome(service=ChromeService(ChromeDriverManager().install()))
        self.driver.maximize_window()

    def closed(self, reason):
        self.driver.quit()

    def parse(self, response):
        # 用互斥锁控制driver的并发访问，避免上下文冲突
        with self.driver_lock:
            self.driver.get(response.url)
            time.sleep(5)

            # 获取总页数
            WebDriverWait(self.driver, 10).until(
                EC.presence_of_element_located((By.CSS_SELECTOR, 'ul.pagination'))
            )
            page_count_selector = "ul.pagination li:last-child"
            page_count = self.driver.find_element(By.CSS_SELECTOR, page_count_selector).text
            try:
                page_count = int(page_count)
            except:
                page_count = 1

            # 遍历每一页
            for i in range(1, page_count + 1):
                WebDriverWait(self.driver, 10).until(
                    EC.presence_of_element_located((By.CSS_SELECTOR, 'ul.pagination'))
                )
                if i != 1:
                    current_page = self.driver.find_element(By.CSS_SELECTOR, 'a#secili-sayfa')
                    next_page = current_page.find_element(By.XPATH, 'following::li/a[@onclick][1]')
                    next_page.click()
                    time.sleep(1)

                # 等待商品元素加载完成
                WebDriverWait(self.driver, 10).until(
                    EC.presence_of_element_located((By.CSS_SELECTOR, 'div.urun-kutusu')))
                time.sleep(2)

                product_elements = self.driver.find_elements(By.CSS_SELECTOR, 'div.urun-kutusu')

                for product_element in product_elements:
                    product_name = product_element.find_element(By.CSS_SELECTOR, 'h2 a').text
                    product_price = product_element.find_element(By.CSS_SELECTOR, 'div.urun-fiyat span').text
                    try:
                        product_price = float(product_price.replace(' TL', '').replace(',', '.'))
                    except:
                        product_price = -1.0
                    product_url = product_element.find_element(By.CSS_SELECTOR,
                                                               'div.kutu-urun-resmi a.kutu-link').get_attribute('href')

                    yield {
                        'name': product_name,
                        'price': product_price,
                        'URL': product_url,
                        'date': datetime.now(),
                    }

关键修改说明 #

• 添加threading.Lock()：通过互斥锁保证同一时间只有一个请求操作Selenium浏览器实例，彻底避免异步请求导致的上下文冲突。
• 简化等待逻辑：移除重复的分页栏等待代码，保留核心的商品元素加载等待，减少不必要的等待时间。
• 资源安全保障：爬虫关闭时正确调用driver.quit()，避免浏览器进程残留。

内容的提问来源于stack exchange，提问作者serkan ertas