Scrapy自定义set_user_agent函数报参数数量错误如何解决

AI 小北

2026-10-10

错误根因

Scrapy 中 Rule 的 process_request 回调方法的参数是固定的：作为类方法首先自带 self 参数，框架还会自动传入request、response两个位置参数。你原代码的set_user_agent只定义了2个参数，框架传入3个参数时就会触发参数数量不匹配的报错。

修复方案

给set_user_agent方法补充接收response参数即可，无需额外使用该参数，修正后的完整代码如下：

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class BestMoviesSpider(CrawlSpider):
    name = 'best_movies'
    allowed_domains = ['imdb.com']
    user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36'

    def start_requests(self):
        yield scrapy.Request(url='https://www.imdb.com/search/title/?genres=drama&groups=top_250&sort=user_rating,desc',
                         headers={
                             # 这里顺便把下划线改成标准的中划线，避免请求头不生效
                             'User-Agent': self.user_agent
                         })

    rules = (
        Rule(LinkExtractor(restrict_xpaths=("//h3[@class='lister-item-header']/a")), callback='parse_item',
            follow=True, process_request='set_user_agent'),
        Rule(LinkExtractor(restrict_xpaths="(//a[@class='lister-page-next next-page'])[2]"),
         process_request='set_user_agent')

 )

    # 新增response参数接收框架传入的对应值
    def set_user_agent(self, request, response):
        request.headers['User-Agent'] = self.user_agent
        return request