使用Python 3.6登录HTTPS网站无法绕过登录页求助

Hey there! Let's tackle this Daxko login problem head-on—grabbing financial transaction data for your nonprofit is important, and platforms like Daxko often have robust authentication systems that can trip up basic scraping scripts. Let’s break down the most common issues and how to fix them:

1. Always Use a Persistent Session #

Daxko relies on session cookies to track authenticated users, so never use one-off requests.get()/post() calls. Instead, create a requests.Session() object to persist cookies across requests:

import requests
from bs4 import BeautifulSoup

# Initialize a persistent session
session = requests.Session()

2. Extract the CSRF Token First #

Nearly all modern login forms use a CSRF (Cross-Site Request Forgery) token to prevent automated requests. You’ll need to fetch the login page first to grab this token:

login_url = "https://operations.daxko.com/Login"

# Fetch the login page to get the CSRF token and session cookies
login_page = session.get(login_url)
soup = BeautifulSoup(login_page.content, "html.parser")

# Find the CSRF token (look for input fields with names like "__RequestVerificationToken")
csrf_token = soup.find("input", {"name": "__RequestVerificationToken"})["value"]

Pro tip: Use your browser’s DevTools (Network tab) to inspect the login form submission—this will show you the exact name of the CSRF token field if it’s different.

3. Match the Exact Form Data from Your Browser #

When you log in manually, your browser sends specific form fields to the server. Use DevTools to capture the POST request and replicate all required fields (don’t just include username/password!):

# Build the login payload using fields from DevTools
login_payload = {
    "Username": "your_nonprofit_username",
    "Password": "your_nonprofit_password",
    "__RequestVerificationToken": csrf_token,
    # Add any other hidden fields you see in the form (e.g., "ReturnUrl", "RememberMe")
    "ReturnUrl": "",
    "RememberMe": "false"
}

4. Add Realistic Request Headers #

Daxko’s servers might block requests without proper headers that mimic a real browser. Include at least these:

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    "Referer": login_url,
    "Content-Type": "application/x-www-form-urlencoded"
}

5. Submit the Login Request and Verify Success #

Send the POST request and check if you’re authenticated. A quick way is to fetch a protected page (like the financial transactions dashboard) and see if the response contains content only logged-in users can access:

# Submit the login request
login_response = session.post(login_url, data=login_payload, headers=headers)

# Verify login by accessing a protected page
protected_url = "https://operations.daxko.com/your-financial-page-path"
protected_page = session.get(protected_url, headers=headers)

# Check if we're logged in (adjust this based on the page's content)
if "Financial Transactions" in protected_page.text:
    print("Login successful! Ready to scrape data.")
else:
    print("Login failed—check your payload, token, or headers.")

Common Pitfalls to Watch For #

• AJAX Login Endpoints: If the login happens via an AJAX call (check DevTools’ XHR/fetch tab), your POST URL won’t be the login page URL—it’ll be an API endpoint like /api/auth/login.
• Two-Factor Authentication (2FA): If your account uses 2FA, you’ll need to handle that (either via SMS/email code retrieval, or use a service like PyOTP if it’s TOTP-based).
• Rate Limiting: Daxko might block you if you send too many requests too quickly. Add delays with time.sleep(1) between requests to avoid getting banned.
• Dynamic Content: If the financial data loads after page load via JavaScript, you might need to use selenium or playwright instead of requests to render the page fully.

内容的提问来源于stack exchange，提问作者Pype