Python 2.7类内函数调用失效：cleanup函数未执行求助

Hey there, let's figure out why your cleanup() method isn't running when you call it from parse(). Here are the most common issues and fixes:

1. Your parse() method might be exiting early due to an exception #

If the code in # do some stuff (before calling cleanup()) throws an error, the program will stop executing before it reaches self.cleanup(). To fix this, wrap your main logic in a try/finally block to ensure cleanup runs no matter what:

def parse(self, response):
    try:
        # Your original "do some stuff" code here
        pass
    finally:
        # This will run even if an error occurs in the try block
        self.cleanup()
    return

2. Verify if cleanup() is actually being called (it might be, but you don't see it) #

Sometimes the method runs, but you don't notice because there's no visible output. Add explicit logging or print statements to confirm execution:

import logging

def parse(self, response):
    # do some stuff
    logging.info("Calling cleanup now...")
    self.cleanup()
    logging.info("Cleanup finished.")
    return

def cleanup(self):
    logging.info("Cleanup function is executing!")
    # Your cleanup logic here

Make sure your Scrapy log level is set to INFO or lower (check your settings.py for the LOG_LEVEL setting) to see these messages.

3. Use Scrapy's spider lifecycle hooks for more reliable cleanup #

Calling cleanup() inside parse() only runs it when that specific parse() method finishes. If your spider has multiple parse methods, or exits without running parse() (e.g., no requests processed), your cleanup won't run. Instead, use Scrapy's spider_closed signal to trigger cleanup when the spider finishes, no matter how it ends:

from scrapy import signals
from scrapy.spiders import Spider

class MySpider(Spider):
    name = "my_spider"

    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        spider = super().from_crawler(crawler, *args, **kwargs)
        # Connect the cleanup method to the spider_closed signal
        crawler.signals.connect(spider.cleanup, signal=signals.spider_closed)
        return spider

    def parse(self, response):
        # do some stuff
        return

    def cleanup(self):
        logging.info("Running cleanup on spider shutdown...")
        # Your cleanup logic here

This way, cleanup() will automatically run when the spider stops—whether it completes all tasks, hits an error, or is manually stopped.

内容的提问来源于stack exchange，提问作者MoreScratch