Scrapy登录成功后访问内页被重定向回首页问题排查

问题描述 #

尝试编写Scrapy程序实现目标站点登录功能，但功能未正常生效，核心爬虫代码如下：

import scrapy
from scrapy.http import FormRequest
from scrapy.utils.response import open_in_browser
class StudentSpider(scrapy.Spider):
    name = "students"

    def start_requests(self):
        login = r"https://iin.stage.edlumina.com"
        return [scrapy.Request(url=login, callback=self.login)]

    def login(self, response):
        token = response.css("form input[name='authenticity_token']::attr(value)").extract_first()
        print("\nToken found:", token)
        formdata = {
            "user[email]": "email",
            "user[password]": "password",
        }
        
        return FormRequest.from_response(response, formdata=formdata, callback = self.scrape_dash)

    def scrape_dash(self, response):
        print("Logged in!")
        print("Heres some user data:", response.body)
        url = r"https://iin.stage.edlumina.com/program_registrations"
        yield scrapy.Request(url=url, callback=self.parse_dash)


    def parse_dash(self, response):
        open_in_browser(response)

程序运行时返回的日志如下：

2022-06-16 11:34:15 [scrapy.utils.log] INFO: Scrapy 2.6.1 started (bot: edlumina)
2022-06-16 11:34:15 [scrapy.utils.log] INFO: Versions: lxml 4.9.0.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 22.4.0, Python 3.10.5 (tags/v3.10.5:f377153, Jun  6 2022, 16:14:13) [MSC v.1929 64 bit (AMD64)], pyOpenSSL 22.0.0 
(OpenSSL 3.0.3 3 May 2022), cryptography 37.0.2, Platform Windows-10-10.0.19044-SP0 
2022-06-16 11:34:15 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'edlumina',
 'NEWSPIDER_MODULE': 'edlumina.spiders',
 'ROBOTSTXT_OBEY': True,
 'SPIDER_MODULES': ['edlumina.spiders']}
2022-06-16 11:34:15 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2022-06-16 11:34:15 [scrapy.extensions.telnet] INFO: Telnet Password: <omitted>
2022-06-16 11:34:15 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2022-06-16 11:34:16 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2022-06-16 11:34:16 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2022-06-16 11:34:16 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2022-06-16 11:34:16 [scrapy.core.engine] INFO: Spider opened
2022-06-16 11:34:16 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-06-16 11:34:16 [scrapy.extensions.telnet] INFO: Telnet console listening on <omitted>
2022-06-16 11:34:16 [filelock] DEBUG: Attempting to acquire lock 1427280154336 on C:\Users\<omitted>\Desktop\Scrapy\venv\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-16 11:34:16 [filelock] DEBUG: Lock 1427280154336 acquired on C:\Users\<omitted>\Desktop\Scrapy\venv\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-16 11:34:16 [filelock] DEBUG: Attempting to release lock 1427280154336 on C:\Users\<omitted>\Desktop\Scrapy\venv\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-16 11:34:16 [filelock] DEBUG: Lock 1427280154336 released on C:\Users\<omitted>\Desktop\Scrapy\venv\lib\site-packages\tldextract\.suffix_cache/publicsuffix.org-tlds\de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2022-06-16 11:34:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://iin.stage.edlumina.com/robots.txt> (referer: None)
2022-06-16 11:34:16 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://iin.stage.edlumina.com> (referer: None)

Token found: <omitted>
2022-06-16 11:34:17 [scrapy.core.engine] DEBUG: Crawled (201) <POST https://iin.stage.edlumina.com/login.json> (referer: https://iin.stage.edlumina.com)
Logged in!
Heres some user data: b'{"user":{"first_name":<omitted>,"last_name":<omitted>,"email":<omitted>,"authentication_token":<omitted>}}'
2022-06-16 11:34:17 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://iin.stage.edlumina.com/> from <GET https://iin.stage.edlumina.com/program_registrations>
2022-06-16 11:34:17 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://iin.stage.edlumina.com/> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2022-06-16 11:34:17 [scrapy.core.engine] INFO: Closing spider (finished)
2022-06-16 11:34:17 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1457,
 'downloader/request_count': 4,
 'downloader/request_method_count/GET': 3,
 'downloader/request_method_count/POST': 1,
 'downloader/response_bytes': 8795,
 'downloader/response_count': 4,
 'downloader/response_status_count/200': 2,
 'downloader/response_status_count/201': 1,
 'downloader/response_status_count/302': 1,
 'dupefilter/filtered': 1,
 'elapsed_time_seconds': 1.145782,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2022, 6, 16, 15, 34, 17, 435784),
 'httpcompression/response_bytes': 15168,
 'httpcompression/response_count': 2,
 'log_count/DEBUG': 10,
 'log_count/INFO': 10,
 'request_depth_max': 2,
 'response_received_count': 3,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 3,
 'scheduler/dequeued/memory': 3,
 'scheduler/enqueued': 3,
 'scheduler/enqueued/memory': 3,
 'start_time': datetime.datetime(2022, 6, 16, 15, 34, 16, 290002)}
2022-06-16 11:34:17 [scrapy.core.engine] INFO: Spider closed (finished)

提交FormRequest后，响应体返回了账号关联信息与authentication_token，初步判断登录已成功。但尝试访问站点内页时，会被自动重定向回首页，该现象与手动删除session cookie后的表现一致，怀疑是cookie或认证令牌未被正确存储导致。需要检查代码是否存在写法问题，同时确认Scrapy是否会自动处理会话cookie，还是需要开发者手动维护。

问题解答 #

• 关于Cookie处理：从日志可以看到CookiesMiddleware已经在默认启用的下载中间件列表中，Scrapy默认会自动维护同域请求的会话Cookie，正常登录流程下不需要开发者手动处理Cookie存储和携带。
• 核心问题定位：日志显示登录POST请求被发送到了/login.json接口，这个是返回JSON格式数据的API端点，只会返回用户信息和接口鉴权token，不会下发网页访问需要的会话Cookie，所以后续访问网页内页时因为没有有效登录凭证被重定向回首页。
  出现这个问题的原因是FormRequest.from_response会自动读取页面表单的action属性作为提交地址，当前页面表单默认配置的提交地址就是JSON接口，不符合网页端登录的需求。
• 代码存在的其他问题：代码中提前提取了Rails站点必备的authenticity_token参数，但没有把这个参数加入formdata提交，部分站点会直接拦截缺失该参数的登录请求。

修复方案 #

修改login方法的表单提交逻辑，手动指定网页端登录端点，同时补全缺失的token参数：

def login(self, response):
    token = response.css("form input[name='authenticity_token']::attr(value)").extract_first()
    print("\nToken found:", token)
    formdata = {
        "user[email]": "实际登录邮箱",
        "user[password]": "实际登录密码",
        "authenticity_token": token
    }
    # 手动指定网页版登录提交地址，不要使用from_response自动识别的json接口
    return FormRequest(
        url="https://iin.stage.edlumina.com/login",
        formdata=formdata,
        callback=self.scrape_dash
    )

修改后可以观察登录POST请求的响应头，如果存在Set-Cookie字段，就说明会话Cookie已经被Scrapy自动存储，后续同域请求会自动携带，不会再出现被重定向回首页的问题。日志中出现的重复请求被过滤是Scrapy去重机制的正常表现，根源是之前请求内页时没有有效凭证被重定向到了已经爬取过的首页。

内容的提问来源于stack exchange，提问作者Klikbait