爬取Daraz.pk时遇TypeError：NoneType对象不可迭代问题求助

问题场景 #

爬取电商网站Daraz.pk时，爬虫触发TypeError: 'NoneType' object is not iterable错误，尝试异常处理未解决，相关代码及报错信息如下：

原代码 #

import scrapy
from selenium.webdriver import Chrome
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from ..items import EcomItem

class DarazSpider(scrapy.Spider):
    name = 'daraz'
    def start_requests(self):
        path = 'C:\Program Files (x86)\chromedriver.exe'
        driver = Chrome(executable_path=path)
        driver.get('https://www.daraz.pk/')
        electronics = driver.find_element(By.NAME, 'q')
        electronics.send_keys('Books')
        electronics.send_keys(Keys.RETURN)
        link_elements = driver.find_elements(By.XPATH,'/html/body/div[3]/div/div[2]/div/div/div/div[2]/div/div/div/div[2]/div[2]/a[text()]')
        for link_el in link_elements:
                    href = link_el.text
                    print(href)
    def parse(self, response):
        pass

报错信息 #

Traceback (most recent call last):
    d = crawler.crawl(*args, **kwargs)
  File "C:\Users\Intag\New folder (2)\lib\site-packages\twisted\internet\defer.py", line 1905, in unwindGenerator
    return _cancellableInlineCallbacks(gen)
  File "C:\Users\Intag\New folder (2)\lib\site-packages\twisted\internet\defer.py", line 1815, in _cancellableInlineCallbacks
    _inlineCallbacks(None, gen, status)
--- <exception caught here> ---
  File "C:\Users\Intag\New folder (2)\lib\site-packages\twisted\internet\defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "C:\Users\Intag\New folder (2)\lib\site-packages\scrapy\crawler.py", line 103, in crawl
    start_requests = iter(self.spider.start_requests())
builtins.TypeError: 'NoneType' object is not iterable
2022-08-06 10:29:20 [twisted] CRITICAL:
Traceback (most recent call last):
  File "C:\Users\Intag\New folder (2)\lib\site-packages\twisted\internet\defer.py", line 1660, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "C:\Users\Intag\New folder (2)\lib\site-packages\scrapy\crawler.py", line 103, in crawl
    start_requests = iter(self.spider.start_requests())
TypeError: 'NoneType' object is not iterable

错误原因及解决方法 #

核心原因 #

Scrapy要求start_requests方法必须返回可迭代的Request对象（比如通过yield生成，或者返回Request列表），但原方法没有任何返回值，默认返回None，导致Scrapy尝试迭代None时触发错误。此外，原代码中获取链接的方式错误（取了元素文本而非href属性）。

修改后的代码 #

import scrapy
from selenium.webdriver import Chrome
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from ..items import EcomItem

class DarazSpider(scrapy.Spider):
    name = 'daraz'
    
    def start_requests(self):
        path = r'C:\Program Files (x86)\chromedriver.exe'  # 加r避免转义问题
        driver = Chrome(executable_path=path)
        try:
            driver.get('https://www.daraz.pk/')
            search_box = driver.find_element(By.NAME, 'q')
            search_box.send_keys('Books')
            search_box.send_keys(Keys.RETURN)
            
            # 改用更健壮的XPATH定位商品链接
            link_elements = driver.find_elements(By.XPATH, '//div[@class="gridItem--Yd0sa"]//a')
            for link_el in link_elements:
                href = link_el.get_attribute('href')  # 获取真实链接而非文本
                if href:  # 过滤空链接
                    yield scrapy.Request(url=href, callback=self.parse)
        finally:
            driver.quit()  # 确保浏览器关闭，避免资源泄漏
    
    def parse(self, response):
        # 在这里处理商品页面数据，示例如下
        item = EcomItem()
        item['title'] = response.xpath('//h1[@class="pdp-mod-product-badge-title"]/text()').get()
        item['price'] = response.xpath('//span[@class="pdp-price pdp-price_type_normal pdp-price_color_orange pdp-price_size_xl"]/text()').get()
        yield item

关键修改点 #

1. 返回Request对象：在start_requests中通过yield scrapy.Request将获取到的商品链接交给Scrapy处理，确保方法返回可迭代对象。
2. 正确获取链接：用get_attribute('href')获取a标签的真实链接，而非text（原代码取的是链接显示文本，不是URL）。
3. 路径转义处理：路径字符串前加r避免反斜杠转义问题。
4. 资源清理：用try...finally确保浏览器会话关闭，防止资源泄漏。
5. 健壮定位：替换脆弱的绝对XPATH为相对XPATH，避免页面结构小变动导致定位失败。

内容的提问来源于stack exchange，提问作者Zain Asif