登录失败后重启Scrapy爬虫的实现与问题解决

我来分享下解决这个问题的完整过程，希望能帮到遇到同样情况的开发者：

初始爬虫实现 #

我写了一个基于InitSpider的ExpireddomainsSpider，核心逻辑是从users.txt随机读取一行，获取账号、密码、代理和UA信息：

class ExpireddomainsSpider(InitSpider): 
    name = "expiredomains" 
    def __init__(self,typ=None, *args, **kwargs): 
        super(ExpireddomainsSpider, self).__init__(*args, **kwargs) 
        self.typ = typ 
        users=open("users.txt","r") 
        dane = self.random_line(users) 
        dane = dane.split(':') 
        self.user = dane[0] 
        self.password = dane[1] 
        self.ip = dane[2] 
        self.port = dane[3] 
        self.headers = {"User-Agent": dane[4]}

同时实现了登录请求和登录状态校验函数：

def login(self, response): 
    self.log("USER: "+self.user+" PASS: "+self.password) 
    return FormRequest('https://member.expireddomains.net/login/', formdata={'login': self.user, 'password': self.password}, callback=self.check_login_response, method='POST')

def check_login_response(self, response): 
    sprawdz = self.user.title() 
    if sprawdz in response.body: 
        self.log("Successfully logged in. Let's start crawling!") 
        return scrapy.FormRequest('http://expireddomains.net/', callback=self.start_crawl) 
    elif "Your account was disabled" in response.body: 
        self.log("Your account was disabled!") 
        super(ExpireddomainsSpider, self).__init__() 
    else: 
        self.log("Bad times :(")

第一次尝试的问题 #

当登录失败（比如账号被禁用）时，我尝试调用super(ExpireddomainsSpider, self).__init__()重新初始化账号信息，但完全没用，爬虫直接就关闭了。

第二次改进：抽离初始化逻辑 #

后来我把账号初始化的逻辑抽成了单独的init()方法，修改后的类结构：

class ExpireddomainsSpider(InitSpider): 
    name = "expiredomains" 
    def init(self): 
        users=open("users.txt","r") 
        dane = self.random_line(users) 
        dane = dane.split(':') 
        self.user = dane[0] 
        self.password = dane[1] 
        self.ip = dane[2] 
        self.port = dane[3] 
        self.headers = {"User-Agent": dane[4]} 
    def __init__(self,typ=None, *args, **kwargs): 
        super(ExpireddomainsSpider, self).__init__(*args, **kwargs) 
        self.typ = typ 
        self.init()

然后修改校验函数，失败时重新调用init()获取新账号，再发起登录请求：

def check_login_response(self, response): 
    sprawdz = self.user.title() 
    if sprawdz in response.body: 
        self.log("Successfully logged in. Let's start crawling!") 
        return scrapy.FormRequest('http://expireddomains.net/', callback=self.start_crawl) 
    elif "Your account was disabled" in response.body: 
        self.log("Your account was disabled!") 
        self.init() 
        return self.login(response) 
    else: 
        self.log("Bad times :(")

但这个方案有个局限：只能重试两次，两次失败后爬虫就直接关闭了，没法持续重试直到拿到可用账号。

最终解决方案：添加dont_filter=True #

问题的根源在于Scrapy的请求去重机制——默认情况下，重复的请求会被过滤掉。我在login函数的FormRequest里加上dont_filter=True参数，让Scrapy不拦截重复的登录请求，这样就能持续重试了：

def login(self, response): 
    return FormRequest(
        'https://member.expireddomains.net/login/', 
        formdata={'login': self.user, 'password': self.password}, 
        callback=self.check_login_response, 
        method='POST',
        dont_filter=True  # 关键参数：禁用请求去重
    )

现在只要users.txt里还有可用账号，爬虫就会一直重试，直到登录成功为止。

内容的提问来源于stack exchange，提问作者Daniel Koczuła