使用requests模块获取JS渲染HTTPS页面响应的可行性咨询

Hey there, let's break down exactly what's going on here and how you can get the content you're expecting.

The Root Cause #

The page you're targeting is a JavaScript-rendered Single-Page Application (SPA)—you can tell from the Angular <ng-view> tag and the loaded .js bundles in your response. When you use requests.get(), you're only fetching the initial HTML skeleton of the page. The actual login form, username/password fields, and other dynamic content are generated by the browser after it executes the JavaScript files included in that skeleton. Since requests doesn't run JavaScript, it can't access the fully rendered content you see in your browser.

Your Options to Fix This #

Option 1: Use a Browser Automation Tool (Selenium/Playwright) #

These tools launch a real browser (Chrome, Firefox, etc.), execute all the page's JavaScript, and let you interact with the fully rendered page. This is the most straightforward way to mimic what your browser does.

Here's a quick example with Selenium (install it first with pip install selenium and download the appropriate browser driver):

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Configure browser to use your client certificate (if required)
options = webdriver.ChromeOptions()
options.add_argument(f'--certificate-file=/etc/pki/tls/mycertfile.pem')
# Add --key-file=/path/to/your/key.pem if your certificate requires a separate private key

# Launch the browser
driver = webdriver.Chrome(options=options)

# Navigate to the target page
driver.get('https://10.22.156.151/performance')

# Wait for the login form elements to load (adjust selectors to match your page's actual IDs/classes)
wait = WebDriverWait(driver, 10)
username_field = wait.until(EC.presence_of_element_located((By.ID, 'username')))
password_field = driver.find_element(By.ID, 'password')
submit_button = driver.find_element(By.ID, 'submit')

# Fill in credentials and submit
username_field.send_keys('your_username')
password_field.send_keys('your_password')
submit_button.click()

# Wait for the post-login page to load and get the full rendered HTML
page_content = driver.page_source
print(page_content)

# Clean up
driver.quit()

Option 2: Reverse-Engineer the Backend API #

Instead of mimicking the browser, you can directly call the API endpoints that the JavaScript uses to load content. Here's how:

1. Open your browser's DevTools (F12), go to the Network tab.
2. Refresh the page and watch for XHR/Fetch requests that load the login form or performance data.
3. Use requests.Session() to maintain your session (including cookies) and replicate these API calls.

Example code:

import requests
import urllib3

# Disable the SubjectAltName warning (optional but cleans up output)
urllib3.disable_warnings(urllib3.exceptions.SubjectAltNameWarning)

cafile = '/etc/pki/tls/mycertfile.pem'
session = requests.Session()

# First, get the initial page to set up cookies and session context
session.get('https://10.22.156.151/performance', verify=cafile)

# Replace with your actual login API endpoint and payload (check DevTools for details)
login_payload = {
    'username': 'your_username',
    'password': 'your_password'
}
login_response = session.post(
    'https://10.22.156.151/api/login',  # Adjust this URL to match the actual login endpoint
    data=login_payload,
    verify=cafile
)

# After successful login, fetch the performance page content
response = session.get('https://10.22.156.151/performance', verify=cafile)
print(response.text)

Quick Note on the Certificate Warning #

The SubjectAltNameWarning you're seeing is just a heads-up that the server's certificate doesn't include a Subject Alternative Name (SAN) field. It's not blocking your request (you got a 200 response), but if you want to suppress it, the urllib3.disable_warnings() line in the second example will take care of that.

内容的提问来源于stack exchange，提问作者Sandeep Lade