选择爬虫登录的正确URL，排查Scrapy登录失败无法爬取问题

It makes total sense that your selectors work in the Scrapy Shell but not in your script—chances are you were logged into the site in your browser when you launched the shell (Scrapy can inherit your browser’s cookies if you use tools like --cookie-jar), but your script isn’t handling the authentication flow properly. Let’s walk through the most reliable ways to add login functionality to your spider:

1. Simulate Form-Based Login (Most Common) #

For sites that use a standard login form, you’ll need to send a POST request with your credentials to the login endpoint. Here’s a concrete implementation:

import scrapy

class TargetSpider(scrapy.Spider):
    name = 'target_spider'
    allowed_domains = ['your-target-site.com']
    target_url = 'https://your-target-site.com/restricted-page'

    def start_requests(self):
        # First, fetch the login page to grab any required CSRF tokens
        yield scrapy.Request(
            url='https://your-target-site.com/login',
            callback=self.handle_login,
            meta={'dont_redirect': True}
        )

    def handle_login(self, response):
        # Extract CSRF token (adjust the selector to match the site's form)
        csrf_token = response.css('input[name="csrfmiddlewaretoken"]::attr(value)').get()
        
        # Send login credentials via POST
        yield scrapy.FormRequest(
            url='https://your-target-site.com/login',
            formdata={
                'username': 'your-actual-username',
                'password': 'your-actual-password',
                'csrfmiddlewaretoken': csrf_token  # Omit if the site doesn't use CSRF
            },
            callback=self.verify_login
        )

    def verify_login(self, response):
        # Check if login succeeded (look for a unique element only visible to logged-in users)
        if response.css('div.welcome-message').get():
            self.logger.info("Login successful! Proceeding to scrape target content.")
            # Now fetch the restricted page
            yield scrapy.Request(url=self.target_url, callback=self.parse)
        else:
            self.logger.error("Login failed! Double-check credentials or form field names.")
            return

    def parse(self, response):
        # Your existing selector logic here (the one that worked in the Shell)
        data = response.css('div.target-content::text').getall()
        yield {'scraped_data': data}

2. Use Browser Cookies (For Complex Auth) #

If the site uses OAuth, 2FA, or has tricky anti-scraping measures, you can export cookies from your logged-in browser and reuse them in your script:

Steps: #

• Use a browser extension (like "Export Cookies" for Chrome/Firefox) to save your cookies as a JSON file.
• Load the cookies in your spider:

import json
import scrapy

class TargetSpider(scrapy.Spider):
    name = 'target_spider'
    allowed_domains = ['your-target-site.com']

    def start_requests(self):
        # Load cookies from exported JSON file
        with open('logged-in-cookies.json', 'r') as f:
            cookies = json.load(f)
        
        # Send request with authenticated cookies
        yield scrapy.Request(
            url='https://your-target-site.com/restricted-page',
            cookies=cookies,
            callback=self.parse
        )

    def parse(self, response):
        # Your existing selector logic here
        pass

3. Debugging Tips to Troubleshoot #

• Print response.status to check if you’re getting a 302 redirect (to login page) or 403 Forbidden.
• Use print(response.body.decode('utf-8')) to see if the response is the login page instead of the restricted content.
• Compare request headers in the Shell (response.request.headers) with your script’s headers—ensure you’re sending the same User-Agent and Referer values.

Always remember to respect the site’s robots.txt and terms of service when scraping!

内容的提问来源于stack exchange，提问作者Melinsuna