Python爬取需登录网站遇问题：登录页路径异常致登录失败

Hey there, let's figure out why your login code isn't working with that /login POST form. I've run into similar issues before, so here are the most likely culprits and fixes to try:

1. You're Missing Hidden Form Fields #

That login form you shared (<form id="login-box" ... action="/login" method="post">) almost certainly has hidden input fields that your code isn't including—things like CSRF tokens, authenticity tokens, or other site-specific values. These are required for the server to accept your login request.

To fix this:

• First, fetch the login page source (using a GET request) and parse out all the <input> tags inside the form. Look for elements like:

  <input type="hidden" name="csrfmiddlewaretoken" value="abc123xyz">
  

• Include every single one of these fields in your POST data, along with your username and password.

2. Your Request Headers Are Incomplete #

Most modern websites validate request headers to block bots. The most critical ones to include are:

• User-Agent: Mimic a real browser (e.g., Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36)
• Referer: Should be the full URL of the login page (e.g., https://your-target-site.com/login)
• If you're using cookies, make sure you're persisting them across requests (more on that below).

3. You're Not Persisting the Session #

When you log in successfully, the server sets session cookies that your subsequent requests need to carry to stay authenticated. If you're using separate get()/post() calls instead of a session object, you'll lose these cookies immediately.

Here's a quick example using Python's requests library with session handling:

import requests
from bs4 import BeautifulSoup

# Initialize a session to persist cookies
session = requests.Session()

# First, fetch the login page to get CSRF token and initial cookies
login_url = "https://your-target-site.com/login"
login_page = session.get(login_url)

# Parse the CSRF token from the page (adjust the selector to match your site)
soup = BeautifulSoup(login_page.text, "html.parser")
csrf_token = soup.find("input", {"name": "csrfmiddlewaretoken"})["value"]

# Build your login data (include ALL form fields, hidden ones too!)
login_payload = {
    "username": "your-username",
    "password": "your-password",
    "csrfmiddlewaretoken": csrf_token
}

# Set up headers to mimic a real browser
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    "Referer": login_url
}

# Send the login POST request
login_response = session.post(login_url, data=login_payload, headers=headers)

# Verify login success (check for a welcome message, redirect, or 200 status with authenticated content)
if "Welcome back" in login_response.text:
    print("Login worked!")
    # Now use the session to access your authorized database
    db_response = session.get("https://your-target-site.com/authorized-db-page")
    # Do something with db_response content
else:
    print("Login failed—check your payload or headers!")

4. You're Using a Relative URL Instead of Absolute #

The form's action="/login" is a relative path. Make sure your POST request is sending data to the full absolute URL (e.g., https://your-target-site.com/login) instead of just /login—otherwise, your request might be going to the wrong endpoint.

5. The Site Encrypts Passwords Before Submission #

Some sites use JavaScript to encrypt or hash passwords before sending them to the server. If this is the case, your plaintext password won't work. To test this:

• Use your browser's DevTools (Network tab) to capture the actual POST data when you log in manually.
• If the password field in the request isn't your plaintext password, you'll need to replicate that encryption logic in your code, or use a tool like Selenium to simulate a real browser login.

Pro Tip: Compare Your Request to a Browser's #

The easiest way to debug this is to:

1. Log in manually using Chrome/Firefox DevTools open to the Network tab.
2. Find the POST request to /login and inspect its Form Data, Headers, and Cookies.
3. Compare that to what your code is sending—any difference is likely the problem.

内容的提问来源于stack exchange，提问作者eleween