Scrapy爬虫运行报TypeError: 'dict' object is not callable如何解决

核心错误原因 #

报错日志中的TypeError: 'dict' object is not callable是因为混淆了字典取值和函数调用的语法：
Scrapy中response.request.meta是字典类型，访问字典元素需要用方括号[]，你使用圆括号()相当于把字典当做函数调用，触发了类型错误。

第二个潜在错误 #

你在info_parse方法中构造跳转书籍详情页的请求时，没有传递meta参数，后续book_info方法尝试读取meta字段时会触发KeyError，也需要同步修复。

具体修改点 #

1. 把所有response.request.meta('xxx')替换为response.request.meta['xxx']
2. 在info_parse的scrapy.Request中补充传递meta参数

修复后完整代码 #

import scrapy
from scrapy.http import HtmlResponse
import requests
from bs4 import BeautifulSoup

class ScrapSpider(scrapy.Spider):
    name = 'scrapp'
    allowed_domains = ['books.toscrape.com']
    start_urls = ['http://books.toscrape.com/']

    def parse(self, response):
        categ=response.xpath('//div[@class="side_categories"]/ul[@class="nav nav-list"]/li/ul/li')
        Category_Name=categ.xpath('.//a[contains(text(),"Historical Fiction")]/text()').get().replace('\n',"").strip()
        Kategorylink=categ.xpath('.//a[contains(text(),"Historical Fiction")]/@href').get().replace('\n',"").strip()
        yield{
           'Category_Name':Category_Name,
           'Kategorylink':response.urljoin(Kategorylink)
        }
        yield scrapy.Request(url=response.urljoin(Kategorylink),callback=self.info_parse,meta={'category_name':Category_Name,'category_link':Kategorylink})  
    
    def info_parse(self,response):
        # 修复字典取值语法
        Category_Name=response.request.meta['category_name']
        Kategorylink=response.request.meta['category_link']      
        Book_Frame=response.xpath('//section/div/ol/li/article[@class="product_pod"]/h3/a/@href')
        for books in Book_Frame:            
            # 补充传递meta参数
            yield scrapy.Request(url=response.urljoin(books),callback=self.book_info, meta={'category_name':Category_Name,'category_link':Kategorylink})           
            

    def book_info(self,response):
        # 修复字典取值语法
        Category_Name=response.request.meta['category_name']
        Kategorylink=response.request.meta['category_link']
        name= response.xpath('//*[@class="price_color"]/text()').get()
        yield{
            'Category_Name':Category_Name,
            'Categorylink':Kategorylink,
            'Books':name

        }

内容的提问来源于stack exchange，提问作者Abu Bakar Siddique