Scrapy回调函数未返回全部结果的问题排查与解决

我是Scrapy新手，想要从service-public.fr网站抓取邮箱地址。当前爬虫设置了CLOSESPIDER_PAGECOUNT为600，不添加callback时能获取约600条结果，但添加callback后仅得到27条结果，请问该如何解决？我考虑过移除callback，但不确定是否正确。

我的Scrapy爬虫代码：

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from scrapy import Field

class CrawlingSpider(CrawlSpider):
    name = "mycrawler"
    allowed_domains = ["service-public.fr"]
    start_urls = ["https://lannuaire.service-public.fr/navigation/mairie?where=71&"]

    COUNT_MAX = 600

    custom_settings = {
        'CLOSESPIDER_PAGECOUNT': COUNT_MAX
    }

    rules = (
        Rule(LinkExtractor(allow="saone-et-loire", deny=["banque_de_france","epide","pole_emploi","mission_local","droit_travail","clic","tribunal_commerce","centre_penitentiaire","laboratoire_departemental","cnftp","ordre_avocats","permanence_juridique","spip","sdjes","onac","ddsp","dmd","commissariat_police","maison_handicapees","sdis","cirfa","prefecture","safer","fr_renov","ddt","sous_pref","point_accueil_numerique","cdg","cg","chambre_notaires","civi","commission_conciliation","bav","fdc","ddva","maia","pmi","sdac","caf","cpam","ddpp","epci","ars_antenne","cij","ad","caue","prefecture_greffe_associations","crib","dd_femmes","demande-de-mise-a-jour","msa","pif","ssti","urssaf","afpa","permanance_juridique","aav","prudhommes","gendarmerie","dreal_ut","anah","greta","canope_atelier","cio","tresorerie","sip","sie","dd_fip","hypotheque","pcb","paierie_departementale","credit_municipal"]), callback="aaa"),
    )

    def aaa(self, response):
        yield {
            "mail": response.css(".send-mail::text").get(),
        }

• 问题核心原因：给Rule加callback后，Scrapy默认不会继续跟进当前页面里的链接（即follow=False），所以爬虫只爬完第一批匹配到的27个页面就停止了，没继续深度抓取后续链接。而不加callback时，默认follow=True，爬虫会持续跟进页面链接直到达到600页的限制。

• 最优修改方案：在Rule里显式设置follow=True，这样爬虫执行完回调提取邮箱后，还会从当前页面提取新链接继续爬取，既保留数据提取逻辑，又能达到预设的页面爬取数量：

rules = (
    Rule(LinkExtractor(allow="saone-et-loire", deny=[...]), callback="aaa", follow=True),
)

• 关于移除callback：移除callback后，爬虫只会抓取页面但不会执行你的邮箱提取逻辑，得到的结果只是页面响应，没有你需要的邮箱数据，所以这不是正确做法。

• 额外优化建议：

  • 你当前的deny列表过于冗长，可以简化规则，比如只聚焦抓取mairie相关页面，减少误拦截概率。
  • 在回调函数里增加空值判断，避免生成无邮箱的无效条目：

def aaa(self, response):
    mail = response.css(".send-mail::text").get()
    if mail:
        yield {"mail": mail.strip()}

内容的提问来源于stack exchange，提问作者Citron