Scrapy登录后发起请求自动登出问题求助

用Scrapy登录oddsportal.com后，发起/results/请求会被自动登出。使用Selenium可正常完成登录后访问该页面的操作，但Scrapy复现始终失败。试过50+Stack Overflow上的解决方案，也在Reddit、GitHub等Scrapy相关论坛提问，均未得到有效解答，仅需明确登录后请求触发登出的原因。

测试脚本 #

Selenium脚本 #

from selenium.webdriver import Chrome, ChromeOptions
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By

# 定义URL
url = 'https://www.oddsportal.com/login/'
username = 'chuky'
password = 'A151515a'
path = r'C:\Users\Glodaris\OneDrive\Desktop\Repo\Scraper\chromedriver.exe'
webdriver_service = Service(path)
options = ChromeOptions()

browser = Chrome(service=webdriver_service, options=options)

browser.get(url)
browser.implicitly_wait(2)
browser.find_element(By.ID, 'onetrust-accept-btn-handler').click()
browser.find_element(By.ID,'login-username1').send_keys(username)
browser.find_element(By.ID,'login-password1').send_keys(password)
browser.implicitly_wait(10)
browser.find_element(By.XPATH,'//*[@id="col-content"]//button[@class="inline-btn-2"]').click()

print('successful login')
browser.implicitly_wait(10)
browser.get('https://www.oddsportal.com/results/')

Scrapy脚本 #

import scrapy
from scrapy.spiders import CrawlSpider
import logging

logger = logging.getLogger(__name__)

class OddsportalSpider(CrawlSpider):
    name = 'oddsportal'
    allowed_domains = ['oddsportal.com']  
    login_page = 'https://www.oddsportal.com/login/'

    def start_requests(self):
        """爬取开始前执行，尝试登录"""
        yield scrapy.Request(
            url=self.login_page,
            callback=self.login,
            dont_filter=True    
        )

    def login(self, response):
        """生成登录请求"""
        yield scrapy.FormRequest.from_response(
            response=response,
            formdata={
                'login-username': 'chuky', 
                'login-password': 'A151515a',
                'login-submit': '',
            },
            callback=self.after_login,
            dont_filter=True
        )

    def after_login(self, response):
        """验证登录结果，成功则发起/results/请求"""
        if b"Wrong username or password" in response.body:
            logger.warning("LOGIN ATTEMPT FAILED")
            return
        else:
            logger.info("LOGIN ATTEMPT SUCCESSFUL")
            url = 'https://www.oddsportal.com/results/'
            return scrapy.Request(url=url, callback=self.parse_item, dont_filter=True) 

    def parse_item(self, response):  
        print('Thissssssssss----------------------', response.url)
        from scrapy.utils.response import open_in_browser
        open_in_browser(response) 

登录成功后发起/results/请求时会被登出。Scrapy默认处理Cookie，但手动在每个请求中附带Cookie和请求头也无效，日志反馈登录成功，但后续请求触发登出。

复现步骤 #

• scrapy startproject oddsportal
• scrapy genspider -t crawl oddsportal oddsportal.com
• 在settings.py中设置默认User-Agent：USER_AGENT = 'oddsportal_website (+http://www.yourdomain.com)'
• 运行爬虫：scrapy crawl oddsportal

日志信息 #

{'BOT_NAME': 'oddsportal_website',
 'DUPEFILTER_CLASS': 'scrapy_splash.SplashAwareDupeFilter',
 'HTTPCACHE_STORAGE': 'scrapy_splash.SplashAwareFSCacheStorage',
 'NEWSPIDER_MODULE': 'oddsportal_website.spiders',
 'ROBOTSTXT_OBEY': True,
 'SPIDER_MODULES': ['oddsportal_website.spiders']}
2022-08-15 09:47:48 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2022-08-15 09:47:48 [scrapy.extensions.telnet] INFO: Telnet Password: 66aa39ca3b133f3d
2022-08-15 09:47:48 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2022-08-15 09:47:48 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'oddsportal_website.middlewares.UserAgentRotatorMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy_splash.SplashCookiesMiddleware',
 'scrapy_splash.SplashMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2022-08-15 09:47:48 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy_splash.SplashDeduplicateArgsMiddleware']
{'log_count/DEBUG': 9,
 'log_count/INFO': 11,
 'request_depth_max': 2,
 'response_received_count': 4,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 3,
 'scheduler/dequeued/memory': 3,
 'scheduler/enqueued': 3,
 'scheduler/enqueued/memory': 3,
 'start_time': datetime.datetime(2022, 8, 15, 8, 47, 48, 449490)}

内容的提问来源于stack exchange，提问作者benjamin olise