求助：Scrapy中CrawlerRunner无控制台日志输出，CrawlerProcess正常

Hey there, I’ve run into this exact issue before! The problem here is that CrawlerRunner is built to be a lightweight tool—it doesn’t automatically set up Scrapy’s logging system for you, unlike CrawlerProcess which handles that out of the box. That’s why you’re not seeing any console logs when using Runner.

Luckily, fixing this is straightforward. You just need to manually configure Scrapy’s logging before initializing the CrawlerRunner. Here’s how:

Quick Fix: Use Scrapy’s Built-in Log Config #

Call Scrapy’s configure_logging() utility function first—it sets up default log formatting, levels, and routes logs to the console, matching CrawlerProcess’s default behavior perfectly.

Example code:

from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings

# This line is the key to enabling console logs!
configure_logging()

# Load your project settings and initialize the Runner
settings = get_project_settings()
runner = CrawlerRunner(settings)

# Add your spider and start the runner
runner.crawl("your_spider_name")
runner.start()

Customize Logging (Optional) #

If you want more control—like adjusting log levels, customizing the output format, or sending logs to a file—you can skip the default root handler and set up your own config:

from scrapy.crawler import CrawlerRunner
from scrapy.utils.log import configure_logging
from scrapy.utils.project import get_project_settings
import logging

# Disable the default root handler to use our custom setup
configure_logging(install_root_handler=False)

# Define your custom logging config
logging.basicConfig(
    level=logging.DEBUG,  # Switch to DEBUG for more detailed logs
    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    handlers=[logging.StreamHandler()]  # Ensures logs go to the console
)

settings = get_project_settings()
runner = CrawlerRunner(settings)
runner.crawl("your_spider_name")
runner.start()

To sum it up: CrawlerRunner gives you flexibility by leaving auxiliary setup (like logging) in your hands, while CrawlerProcess is a "batteries-included" option that handles these steps automatically. Adding that single configure_logging() call bridges the gap.

内容的提问来源于stack exchange，提问作者Aerodynamic