求助：使用Scrapy登录vBulletin论坛失败，需详细指导

Hey there! Let's work through this step by step—vBulletin login systems can be finicky, but we'll take it slow so you can follow along as someone new to scraping.

vBulletin relies on a few key things to let you log in, so we first need to "spy" on what your browser sends when you log in manually. Here's how to do that with Chrome (other browsers work similarly):

• Open your company forum's login page, then press F12 to open the Developer Tools.
• Click the Network tab at the top, then check the box that says Preserve log (so we don't lose the login request after redirect).
• Manually log in to the forum like you normally do.
• In the Network tab, look for a POST request (it might be named login.php or have do=login in the URL). Click on it.
  • Go to the Payload (or Form Data) section: this shows all the data your browser sent to log in—usernames, passwords, hidden security tokens, etc. Write these down (you'll need them later).
  • Go to the Headers section: copy the User-Agent string (this tells the site you're using a real browser, not a scraper) and note any cookies listed.

If you haven't already, install Scrapy first (run pip install scrapy in your command prompt/terminal). Then:

1. Create a new Scrapy project:

   scrapy startproject forum_scraper
   cd forum_scraper
   

2. Open the spiders folder inside your project, create a new file called forum_login_spider.py, and paste the code below. I'll explain every part so you know what to change:

import scrapy
from scrapy.http import FormRequest
# Uncomment this if your forum uses MD5-hashed passwords (we'll talk about this later)
# import hashlib

class ForumLoginSpider(scrapy.Spider):
    name = "forum_login"
    # Replace this with your actual forum login URL
    start_urls = ["https://your-company-forum.com/login.php"]

    def parse(self, response):
        # First, grab the security token (vBulletin uses this to prevent fake login requests)
        # Use the name attribute from the hidden input you saw in the Form Data earlier
        security_token = response.xpath('//input[@name="securitytoken"]/@value').get()

        # Build the login form data using what you copied from the Network tab
        formdata = {
            'vb_login_username': 'YOUR_USERNAME_HERE',  # Replace with your actual username
            'vb_login_password': 'YOUR_PASSWORD_HERE',  # Replace with your actual password
            'securitytoken': security_token,
            'do': 'login',  # This is almost always present in vBulletin login requests
            # Add any other fields you saw in Form Data here (like vb_login_md5password if needed)
        }

        # If your forum uses MD5-hashed passwords instead of plain text, replace the password line above with:
        # password_md5 = hashlib.md5("YOUR_PASSWORD_HERE".encode('utf-8')).hexdigest()
        # formdata['vb_login_md5password'] = password_md5
        # formdata['vb_login_password'] = ''  # Leave this empty if MD5 is used

        # Send the login request
        yield FormRequest(
            url=response.url,  # Use the login URL (some forums use a separate processing URL—check your Network tab!)
            formdata=formdata,
            callback=self.after_login,
            headers={
                # Replace this with the User-Agent you copied from the Headers tab
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
                'Referer': response.url  # Tells the site you came from the login page
            }
        )

    def after_login(self, response):
        # Check if login worked—look for an element only visible when logged in (like your username)
        # Adjust the XPath to match your forum's HTML (use F12 to find the right selector)
        if response.xpath('//span[@class="username"]/text()').get():
            self.logger.info("✅ Login successful!")
            # Now go to the forum section you want to scrape
            yield scrapy.Request(url="https://your-company-forum.com/forum-section-url", callback=self.parse_forum)
        else:
            self.logger.error("❌ Login failed! Double-check your username/password and form fields.")

    def parse_forum(self, response):
        # This is where you'll scrape the forum content—example below extracts post titles and links
        for post in response.xpath('//a[@class="threadtitle"]'):
            yield {
                'Post Title': post.xpath('./text()').get(),
                'Post Link': post.xpath('./@href').get()
            }

        # Handle pagination (if your forum has multiple pages of posts)
        next_page = response.xpath('//a[@title="Next Page"]/@href').get()
        if next_page:
            yield scrapy.Request(url=next_page, callback=self.parse_forum)

Once you've updated the code with your forum's details:

1. Open your command prompt/terminal, navigate to your project folder (cd forum_scraper).
2. Run the scraper and export directly to Excel:

   scrapy crawl forum_login -o forum_posts.xlsx
   

Scrapy will automatically save all the data you scraped into an Excel file in your project folder—no extra work needed!

• Incorrect Security Token: Always extract the token from the login page (don't hardcode it)—vBulletin generates a new token for each session.
• User-Agent Blocking: If you get blocked, make sure you're using a real browser's User-Agent (not Scrapy's default one).
• Password Encryption: Some older vBulletin versions use MD5 hashed passwords—check your Form Data to see if vb_login_md5password is present instead of vb_login_password.
• Captchas: If your forum uses captchas for login, you might need to ask your IT team for a scraper-friendly account (bypassing captchas is hard for beginners).

If you still run into issues, feel free to share the redacted Form Data from your login request (hide your username/password!) and a snippet of the login page's HTML, and we can troubleshoot further.

内容的提问来源于stack exchange，提问作者Svetlozar Draganov