如何通过自建代理服务器自动提交HTML表单登录凭证访问目标网站？

Alright, let's walk through exactly how to build this proxy server that handles form-based login automatically for your target site. I'll break down the implementation step by step using Python, and also clarify how this differs from just using the requests library directly.

Here's how the proxy will work:

• When a client sends a request to the proxy, the proxy first checks if it has an active, authenticated session with the target site.
• If no valid session exists, the proxy will automatically submit the login form with your credentials to get a valid session cookie.
• The proxy then forwards the client's request to the target site using this authenticated session, and sends the target's response back to the client.
• Clients (like your browser or other apps) don't need to handle login at all—they just route traffic through the proxy.

First, install the libraries we'll need:

pip install requests flask beautifulsoup4

• requests: Handles the login flow and forwarding requests to the target site.
• flask: Makes it easy to spin up a lightweight HTTP proxy server.
• beautifulsoup4: Parses the login page to extract hidden fields like CSRF tokens (critical for most modern form-based logins).

1. Build the Login Logic (Get Authenticated Session) #

First, we need a function that logs into the target site and returns a requests.Session object with active login cookies. You'll need to inspect the target site's login form to get the right field names (use your browser's DevTools > Network tab to check the form data sent during login).

import requests
from bs4 import BeautifulSoup

def get_authenticated_session(login_url, username, password):
    # Create a session to persist cookies
    session = requests.Session()
    
    # First, fetch the login page to get any required hidden fields (like CSRF token)
    login_page_response = session.get(login_url)
    soup = BeautifulSoup(login_page_response.content, "html.parser")
    
    # Extract CSRF token (adjust the selector to match your target site's field name)
    csrf_token = soup.find("input", {"name": "csrfmiddlewaretoken"})["value"]
    
    # Build the login form data—match all fields from the site's login form
    login_payload = {
        "username": username,
        "password": password,
        "csrfmiddlewaretoken": csrf_token,
        "next": "/"  # Some sites use this to redirect after login; adjust as needed
    }
    
    # Submit the login request
    login_response = session.post(
        login_url,
        data=login_payload,
        headers={"Referer": login_url}  # Most sites require a Referer header for login
    )
    
    # Verify login success (customize this check based on the target site's response)
    if login_response.status_code == 200 and "Welcome" in login_response.text:
        return session
    else:
        raise Exception(f"Login failed. Check credentials or form fields. Response: {login_response.text[:200]}")

2. Build the Proxy Server #

Next, we'll use Flask to create a proxy that uses our authenticated session to forward requests. The proxy will handle all HTTP methods (GET, POST, etc.) and pass through headers/body correctly.

from flask import Flask, request, make_response

app = Flask(__name__)

# Configuration—replace these with your actual values
TARGET_BASE_URL = "http://website.com"
LOGIN_URL = f"{TARGET_BASE_URL}/login/"
USERNAME = "your_username_here"
PASSWORD = "your_password_here"

# Store the authenticated session globally (for simplicity; in production, use a session store)
authenticated_session = None

# Initialize login when the proxy starts
@app.before_first_request
def init_authentication():
    global authenticated_session
    try:
        authenticated_session = get_authenticated_session(LOGIN_URL, USERNAME, PASSWORD)
        print("Successfully authenticated with target site!")
    except Exception as e:
        print(f"Authentication failed: {str(e)}")

# Proxy route that handles all incoming requests
@app.route("/", defaults={"path": ""}, methods=["GET", "POST", "PUT", "DELETE"])
@app.route("/<path:path>", methods=["GET", "POST", "PUT", "DELETE"])
def proxy_request(path):
    if not authenticated_session:
        return "Proxy is not authenticated with the target site.", 500
    
    # Build the full target URL
    target_url = f"{TARGET_BASE_URL}/{path}"
    
    # Extract request details from the client
    request_method = request.method
    request_params = request.args.to_dict()
    request_body = request.get_data() if request_method in ["POST", "PUT"] else None
    
    # Prepare headers to forward—exclude headers that might cause conflicts
    forward_headers = {
        key: value for key, value in request.headers.items()
        if key.lower() not in ["host", "cookie", "content-length"]
    }
    # Add the authenticated session's cookies to the forwarded request
    forward_headers["Cookie"] = "; ".join([f"{name}={value}" for name, value in authenticated_session.cookies.items()])
    
    # Forward the request to the target site
    target_response = authenticated_session.request(
        method=request_method,
        url=target_url,
        params=request_params,
        data=request_body,
        headers=forward_headers,
        allow_redirects=False
    )
    
    # Build the response to send back to the client
    proxy_response = make_response(target_response.content, target_response.status_code)
    # Copy headers from the target site, excluding ones that break the proxy
    for header_name, header_value in target_response.headers.items():
        if header_name.lower() not in ["transfer-encoding", "connection"]:
            proxy_response.headers[header_name] = header_value
    
    return proxy_response

if __name__ == "__main__":
    # Run the proxy on all interfaces, port 8080
    app.run(host="0.0.0.0", port=8080, debug=True)

3. Key Notes & Improvements #

• Form Field Accuracy: Always double-check the login form fields using your browser's DevTools. Missing fields (like CSRF tokens) will cause login failures.
• Session Expiry: Add logic to automatically re-authenticate if the session expires (e.g., check for a 401/403 response from the target site and call get_authenticated_session again).
• Security: In production, don't hardcode credentials—use environment variables or a secure secrets manager. Also, consider adding authentication for the proxy itself so only trusted clients can use it.
• vs. Direct Requests: If you only need to access the site programmatically for your own scripts, using requests.Session() directly is simpler. The proxy is useful when you need to let other clients (browsers, third-party apps) access the site without handling login themselves.

1. Run the proxy script—you should see a message confirming successful authentication.
2. Configure your browser or app to use http://localhost:8080 as the HTTP proxy.
3. Navigate to any page on http://website.com—you'll be automatically logged in without entering credentials.

内容的提问来源于stack exchange，提问作者letroot