如何用Mechanize爬取含无明确URL弹窗登录的网站？

Got it, let's figure out how to handle this popup login scenario with Mechanize. I’ve dealt with similar cases before, so here’s a breakdown of the common approaches:

Most "login popups" on modern sites aren’t actual browser-level popups—they’re just HTML forms hidden with CSS or JavaScript. Since Mechanize ignores CSS/JS visibility rules, you can still access these forms directly from the page source.

Here’s how to approach this:

• Start by visiting the page that triggers the login popup (usually the site’s homepage or a content page that requires authentication).
• List all available forms on the page to identify the login form (it might not be the first one, so don’t just rely on nr=0).
• Select the form by its name, id, or action attribute, then fill in your credentials and submit.

Example code:

import mechanize

br = mechanize.Browser()
br.set_handle_robots(False)

# Visit the page where the login popup appears (e.g., the homepage)
br.open("https://decanter.com")

# List all forms to find the login one (print details to inspect)
print("Available forms:")
for idx, form in enumerate(br.forms()):
    print(f"Form {idx}: Name='{form.name}', Action='{form.action}'")

# Select the login form (adjust based on your inspection—use name, action, or index)
# br.select_form(name="login-form")
# br.select_form(action="/user/login")
br.select_form(nr=1)  # Replace with the correct index if needed

# Fill in credentials (double-check the field names match the form's inputs!)
br['username'] = "myid"  # Might be 'email' or 'userid'—check the form's input names
br['password'] = "mypasswd"

# Submit the form
logged_in_response = br.submit()

# Verify login success (check response content or cookies)
print("Login response status:", logged_in_response.getcode())
print("Current cookies:", br.cookies)

Mechanize doesn’t execute JavaScript, so if the login form only loads after you click a "Login" button (via AJAX), you’ll need to:

• Use your browser’s developer tools (F12 → Network tab) to capture the actual login request. When you click the login button in your browser, look for an XHR/Fetch request sent to an API endpoint (e.g., https://decanter.com/api/auth/login).
• Note the request method (POST is common), the parameters sent (e.g., username, password, maybe a CSRF token), and any required headers or cookies.

Then, use Mechanize to send a direct request to that endpoint:

import mechanize
import urllib.parse

br = mechanize.Browser()
br.set_handle_robots(False)

# First, visit the site to get necessary session cookies (many sites require this)
br.open("https://decanter.com")

# Construct the login payload (match the parameters you saw in the dev tools)
login_payload = {
    "username": "myid",
    "password": "mypasswd",
    "csrf_token": br.form.find_control("csrf_token").value  # Extract CSRF if required
}

# Encode the payload and send the POST request to the login API
encoded_payload = urllib.parse.urlencode(login_payload).encode('utf-8')
logged_in_response = br.open("https://decanter.com/api/auth/login", data=encoded_payload)

# Verify success
print("Login API response:", logged_in_response.read().decode('utf-8'))

If the login is a native browser popup (like alert() or confirm()—though this is extremely uncommon for login flows), Mechanize can’t handle it. In this case, you’d need to switch to a tool that simulates a real browser, like Selenium, which can interact with JavaScript-driven popups.

Pro Tips #

• Use br.view_source() to print the full page HTML and confirm if the login form exists in the source.
• Always double-check the form’s input field names (they might not be userid/passwd—look for name attributes on the input tags).
• Some sites require a CSRF token to prevent cross-site attacks. You’ll need to extract this token from the page source (e.g., from a hidden input) and include it in your login payload.

内容的提问来源于stack exchange，提问作者shenglih