如何用Selenium捕获中间脚本触发的指定POST请求的JSON数据？

Problem Context #

You have this Selenium script to trigger a search on the target site:

from selenium import webdriver
base_url = 'https://www.botoxcosmetic.com/women/find-a-botox-cosmetic-specialist'
driver = webdriver.Chrome()
driver.find_element_by_class_name('normalZip').send_keys('10022')
driver.find_element_by_class_name('normalSearch').click()

Your goal is to capture the JSON response from the POST request to https://www.botoxcosmetic.com/sc/api/findclinic/FindSpecialists (triggered by the search click)—without directly sending requests to that URL via Selenium or the requests library.

Optimal Solution: Use Selenium 4 DevTools API #

Selenium 4 introduced native support for the Chrome DevTools Protocol (CDP), which lets you listen to network traffic directly from the driver. This is the cleanest, most integrated approach: it doesn’t require external tools, and captures exactly the request the page’s own scripts send.

Step-by-Step Implementation #

1. Ensure you’re using Selenium 4+: Older versions lack built-in DevTools support.
2. Enable network monitoring via CDP.
3. Listen for completed requests and filter for your target URL/method.
4. Extract and decode the JSON response (CDP returns response bodies as base64-encoded strings).

Full Working Code #

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
import json
import base64

def capture_target_response():
    # Initialize Chrome driver with DevTools support
    driver = webdriver.Chrome(service=Service())
    driver.get('https://www.botoxcosmetic.com/women/find-a-botox-cosmetic-specialist')

    # Enable network monitoring through DevTools
    driver.execute_cdp_cmd('Network.enable', {})
    target_response = None

    def handle_request_finished(request):
        nonlocal target_response
        # Filter for the specific POST request we need
        if (request['request']['url'] == 'https://www.botoxcosmetic.com/sc/api/findclinic/FindSpecialists'
            and request['request']['method'] == 'POST'):
            # Fetch the response body (returned as base64)
            body_data = driver.execute_cdp_cmd('Network.getResponseBody', {'requestId': request['requestId']})
            # Decode base64 and parse to JSON
            decoded_body = base64.b64decode(body_data['body']).decode('utf-8')
            target_response = json.loads(decoded_body)

    # Register the listener for when requests finish
    driver.add_cdp_listener('Network.requestFinished', handle_request_finished)

    # Perform the search to trigger the target POST request
    driver.find_element(By.CLASS_NAME, 'normalZip').send_keys('10022')
    driver.find_element(By.CLASS_NAME, 'normalSearch').click()

    # Wait for the request to complete (adjust timeout as needed)
    driver.implicitly_wait(5)

    # Clean up network monitoring and close the driver
    driver.execute_cdp_cmd('Network.disable', {})
    driver.quit()

    return target_response

# Run the function and print the captured JSON
response_data = capture_target_response()
if response_data:
    print(json.dumps(response_data, indent=2))
else:
    print("Target request was not captured.")

Key Details #

• nonlocal keyword: Lets us modify the target_response variable inside the callback function.
• Base64 decoding: CDP uses base64 to handle binary data in responses, so we decode it before parsing to JSON.
• Modern Selenium syntax: Replaced deprecated find_element_by_class_name with By.CLASS_NAME for compatibility with Selenium 4+.
• Implicit wait: Gives the network request time to complete after clicking search—you can use explicit waits for more precision if needed.

Alternative: Proxy-Based Interception (Less Optimal) #

If you’re stuck on an older Selenium version, you could use a proxy like mitmproxy to intercept traffic. However, this requires setting up and running the proxy separately, adding extra complexity—hence why the DevTools approach is strongly preferred.

内容的提问来源于stack exchange，提问作者ikemblem