如何在Jupyter Notebook中使用Scrapy？爬虫代码无输出求助

问题背景 #

我正在学习网页爬取，在Jupyter Notebook中运行以下Scrapy爬虫代码后未得到任何输出，需要了解如何在Jupyter Notebook中正确使用Scrapy并解决该问题。

爬虫代码 #

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class BooksCrawlSpider(CrawlSpider):
    name = 'books_crawl'
    allowed_domains = ['books.toscrape.com']
    start_urls = ['https://books.toscrape.com/catalogue/category/books/sequential-art_5/page-1.html']

    le_book_details = LinkExtractor(restrict_css='h3 > a')
    le_next = LinkExtractor(restrict_css='.next > a')  # 下一页按钮
    le_cats = LinkExtractor(restrict_css='.side_categories > ul > li > ul > li a')  # 分类链接

    rule_book_details = Rule(le_book_details, callback='parse_item', follow=False)
    rule_next = Rule(le_next, follow=True)
    rule_cats = Rule(le_cats, follow=True)

    rules = (
        rule_book_details,
        rule_next,
        rule_cats
    )

    def parse_item(self, response):
        yield {
            'Title': response.css('h1 ::text').get(),
            'Category': response.xpath('//ul[@class="breadcrumb"]/li[last()-1]/a/text()').get(),
            'Link': response.url
        }

无输出情况 #

解决步骤 #

1. 核心原因：未启动Scrapy爬虫引擎 #

Scrapy本质是命令行工具，仅定义爬虫类不会自动执行。在Jupyter Notebook中，需要手动通过CrawlerProcess启动爬虫引擎。

2. 完整可运行代码示例 #

在原爬虫代码后添加启动逻辑，可选择直接打印输出或保存到文件：

方式一：直接在Jupyter中打印爬取结果

from scrapy.crawler import CrawlerProcess
from scrapy.utils.log import configure_logging

# 配置日志，减少Jupyter中的冗余输出
configure_logging(install_root_handler=False)
import logging
logging.basicConfig(level=logging.INFO)

# 用于收集爬取到的内容
scraped_items = []

class BooksCrawlSpider(CrawlSpider):
    name = 'books_crawl'
    allowed_domains = ['books.toscrape.com']
    start_urls = ['https://books.toscrape.com/catalogue/category/books/sequential-art_5/page-1.html']

    le_book_details = LinkExtractor(restrict_css='h3 > a')
    le_next = LinkExtractor(restrict_css='.next > a')
    le_cats = LinkExtractor(restrict_css='.side_categories > ul > li > ul > li a')

    rule_book_details = Rule(le_book_details, callback='parse_item', follow=False)
    rule_next = Rule(le_next, follow=True)
    rule_cats = Rule(le_cats, follow=True)

    rules = (
        rule_book_details,
        rule_next,
        rule_cats
    )

    def parse_item(self, response):
        item = {
            'Title': response.css('h1 ::text').get(),
            'Category': response.xpath('//ul[@class="breadcrumb"]/li[last()-1]/a/text()').get(),
            'Link': response.url
        }
        scraped_items.append(item)
        yield item

# 初始化爬虫进程
process = CrawlerProcess(settings={
    'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
})

# 启动爬虫
process.crawl(BooksCrawlSpider)
process.start()

# 爬虫结束后打印结果
for item in scraped_items:
    print(item)

方式二：将结果保存到JSON文件

from scrapy.crawler import CrawlerProcess

process = CrawlerProcess(settings={
    'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
    # 配置输出文件
    'FEEDS': {
        'books_data.json': {'format': 'json', 'encoding': 'utf-8'},
    },
})

process.crawl(BooksCrawlSpider)
process.start()

3. 其他注意事项 #

• 确保Jupyter环境已安装Scrapy，若未安装可执行!pip install scrapy（Jupyter中加!执行命令行）。
• 检查LinkExtractor的选择器是否正确，若爬取结果不符合预期，可通过response.css()或response.xpath()在Jupyter中单独测试选择器有效性。

内容的提问来源于stack exchange，提问作者Mahmoud Badr