Scrapy请求对象变更问题及登录爬虫代码技术咨询

Let's break down your issue and fix the incomplete login spider step by step, along with addressing the request object change concerns you're facing:

First: Fix the Obvious Issues in Your Truncated Code #

Your current code has a few immediate problems that need resolving before tackling the request object changes:

• Redundant imports: You've imported FormRequest twice — remove one instance to clean up the code.
• Conflicting init_request and start_requests: In CrawlSpider, overriding start_requests already takes precedence, so your init_request method is redundant and causing a potential loop (it calls start_requests which loops back to the login page). Remove init_request entirely.
• Truncated login method: The code cuts off at if response.css("#ca... — we'll need to complete this to handle form submission properly.
• Deprecated HtmlXPathSelector: This class is no longer needed in modern Scrapy versions; use response.xpath() or response.css() directly instead.

Addressing Request Object Change Concerns #

If you're seeing unexpected behavior with Scrapy's Request/Response objects, here are the most common version-related changes to check:

• Response selector updates: Old code using HtmlXPathSelector(response) should be replaced with response.xpath() or response.css() directly (these methods are built into the Response object now).
• FormRequest parameters: Ensure you're using the correct form field names (match the name attribute of the login form inputs, not their id).
• Request filtering: If you're reusing URLs (like the login page), add dont_filter=True to avoid Scrapy dropping duplicate requests.

Full Corrected Spider Code #

Here's a complete, fixed version of your spider with explanations:

from scrapy.http import Request, FormRequest
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
import subprocess

class LoginSpider(CrawlSpider):
    name = 'loginspider'
    login_page = 'http://145.100.108.148/login5/login.php'
    start_urls = ['http://145.100.108.148/login5/index.php']
    url = 'http://145.100.108.148'
    username = 'test@hotmail.com'
    password = 'test'

    # Define rules for post-login crawling (adjust based on your needs)
    rules = (
        Rule(LinkExtractor(allow=r'/login5/.*'), callback='parse_item', follow=True),
    )

    def start_requests(self):
        # Start by sending a request to the login page
        yield Request(
            url=self.login_page,
            callback=self.login,
            dont_filter=True
        )

    def login(self, response):
        print('\n Response object here')
        print(response)
        
        # Complete the truncated selector logic (example assumes CSRF token in #csrf_token)
        # Replace with the actual selector for your login form's CSRF token or required fields
        csrf_token = response.css('#csrf_token::attr(value)').get()
        
        if csrf_token:
            # Submit the login form with credentials
            yield FormRequest(
                url=self.login_page,
                formdata={
                    'username': self.username,
                    'password': self.password,
                    'csrf_token': csrf_token  # Match your form's actual field name
                },
                callback=self.verify_login
            )
        else:
            self.logger.error("Failed to extract CSRF token from login page")

    def verify_login(self, response):
        # Check if login was successful (adjust the condition to match your target site)
        if "Welcome" in response.text or self.username in response.text:
            self.logger.info("Login successful! Starting crawl...")
            # Proceed to crawl the start_urls after login
            for url in self.start_urls:
                yield Request(url=url, dont_filter=True)
        else:
            self.logger.error("Login failed! Check username/password or form field names")

    def parse_item(self, response):
        # Add your item parsing logic here
        self.logger.info(f"Parsing page: {response.url}")
        # Example: Extract page title
        title = response.css('title::text').get()
        yield {'page_title': title, 'url': response.url}

Key Notes for Scrapy Request/Response Handling #

• Always inspect the login form in your browser's dev tools to confirm field names (right-click the form input → Inspect to get the name attribute).
• Use self.logger instead of print() for better logging integration with Scrapy.
• If you're using an older Scrapy version, double-check the Request/FormRequest parameters (most modern versions are backward compatible, but niche parameters may have changed).

内容的提问来源于stack exchange，提问作者Kevin C