如何用Python登录网站并爬取数据？含HAC成绩查询登录求助

Hey there! Let's break down how to solve your login issue with the HAC portal and cover general web scraping login best practices too.

The https://hac.chicousd.org/ site uses ASP.NET, which relies on dynamic hidden fields like __VIEWSTATE and __EVENTVALIDATION to handle sessions—this is probably why your initial attempts with requests/urllib failed. You need to first fetch these fields from the login page before submitting your credentials. Here's a working example:

Step-by-Step Code #

import requests
from bs4 import BeautifulSoup

# Initialize a session to persist cookies across requests
session = requests.Session()

# Set a realistic user-agent to avoid being flagged as a bot
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
})

login_url = "https://hac.chicousd.org/LoginParent.aspx?page=Default.aspx"

# 1. Fetch the login page to extract hidden ASP.NET fields
response = session.get(login_url)
soup = BeautifulSoup(response.text, "html.parser")

# Extract dynamic hidden fields (verify these names match the page source!)
view_state = soup.find("input", {"name": "__VIEWSTATE"})["value"]
view_state_generator = soup.find("input", {"name": "__VIEWSTATEGENERATOR"})["value"]
event_validation = soup.find("input", {"name": "__EVENTVALIDATION"})["value"]

# 2. Construct the login form data (replace with your actual credentials)
login_payload = {
    "__VIEWSTATE": view_state,
    "__VIEWSTATEGENERATOR": view_state_generator,
    "__EVENTVALIDATION": event_validation,
    "ctl00$ContentPlaceHolder1$txtUserName": "YOUR_USERNAME",
    "ctl00$ContentPlaceHolder1$txtPassword": "YOUR_PASSWORD",
    "ctl00$ContentPlaceHolder1$btnLogin": "Login"  # Match the login button's value from the page
}

# 3. Submit the login request
login_response = session.post(login_url, data=login_payload)

# 4. Verify login success (check for a keyword that appears post-login, like "Dashboard")
if "Dashboard" in login_response.text:
    print("✅ Login successful!")
    # Now you can fetch grades using the same session
    grades_url = "https://hac.chicousd.org/[YOUR_GRADES_PAGE_URL]"
    grades_response = session.get(grades_url)
    # Parse grades with BeautifulSoup here
    grades_soup = BeautifulSoup(grades_response.text, "html.parser")
    # Example: Extract all grade rows
    grade_rows = grades_soup.find_all("tr", class_="some-grade-class")
    for row in grade_rows:
        print(row.text.strip())
else:
    print("❌ Login failed—double-check form field names or credentials!")

Important Notes for HAC: #

• If the form field names (like ctl00$ContentPlaceHolder1$txtUserName) don't work, right-click the username/password inputs on the login page, select "Inspect", and copy their exact name attributes.
• Some school portals might have additional security checks (like IP restrictions or CAPTCHAs). If you hit a CAPTCHA, you may need to use a tool like Selenium to simulate manual input.

Here's a reusable workflow for almost any website:

1. Analyze the Login Flow with Browser Dev Tools #

• Open your browser's DevTools (F12), go to the Network tab, and check "Preserve Log".
• Manually log in to the site, then look at the POST request sent to the login endpoint.
  • Note the Request URL, Form Data (all fields, including hidden ones), and Request Headers (like User-Agent, Referer).

2. Simulate the Login with requests.Session #

• Use requests.Session() to automatically handle cookies (critical for maintaining login sessions).
• First, fetch the login page to extract any dynamic tokens (CSRF, __VIEWSTATE, etc.).
• Build a payload with all required fields (credentials + dynamic tokens) and send a POST request to the login URL.

3. Validate Login Success #

• Access a page that requires authentication (like a dashboard or grades page).
• Check if the response contains content that's only visible to logged-in users (e.g., a username, dashboard title).

4. Scrape Data #

• Use the same session to fetch target pages and parse the HTML with libraries like BeautifulSoup or lxml.
• For sites that load data dynamically with JavaScript, use tools like Selenium or Playwright to simulate a real browser (instead of requests).

Key Best Practices #

• Always use a realistic User-Agent to avoid being blocked.
• Add delays between requests (time.sleep(2)) to mimic human behavior.
• Respect the site's robots.txt and terms of service—don't overload their servers.
• If you encounter CAPTCHAs, consider manual input or third-party CAPTCHA-solving services (use ethically!).

内容的提问来源于stack exchange，提问作者Onkar Sandhu