使用Selenium WebDriver抓取未显示在innerHTML中的页面数据

The issue here is that your target content is loaded dynamically via Knockout.js (evidenced by the data-bind="html:resultsContent" attribute on divresults). Selenium is fetching the element before the framework has finished rendering content into the DOM, which is why you're seeing an empty innerHTML even though the browser dev tools show content. Here are several practical fixes to try:

1. Wait for Dynamic Content to Load (Most Reliable) #

Use WebDriverWait with expected conditions to wait until your target element (or content inside divresults) is present in the DOM. This ensures you don’t try to access content before it’s fully rendered.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

# Wait up to 10 seconds for the target span to appear
wait = WebDriverWait(driver, 10)
wait.until(
    EC.presence_of_element_located((By.CSS_SELECTOR, "span.glyphicon.glyphicon-tasks"))
)

# Now fetch the full innerHTML of divresults
div_results = driver.find_element_by_id("divresults")
print(div_results.get_attribute('innerHTML'))

If you don’t have a specific target element, you can wait for divresults to have non-empty content:

wait.until(
    lambda driver: len(driver.find_element_by_id("divresults").get_attribute('innerHTML').strip()) > 0
)

2. Check if Content is Inside an iframe #

Sometimes dynamic content lives inside an iframe. If that’s the case, switch to the iframe first before accessing elements:

# Locate and switch to the iframe (replace with your iframe's ID/selector)
iframe = driver.find_element_by_id("target-iframe")
driver.switch_to.frame(iframe)

# Now you can access divresults and its content
div_results = driver.find_element_by_id("divresults")
print(div_results.get_attribute('innerHTML'))

# Switch back to the main document when done
driver.switch_to.default_content()

3. Trigger Scroll or User Interaction #

Some content only loads when the element is in view. Try scrolling to the divresults element to trigger content loading:

import time

div_results = driver.find_element_by_id("divresults")
# Scroll the element into view using JavaScript
driver.execute_script("arguments[0].scrollIntoView({behavior: 'smooth', block: 'center'});", div_results)

# Wait a moment for content to load (use WebDriverWait instead of sleep for better reliability)
time.sleep(2)

print(div_results.get_attribute('innerHTML'))

4. Directly Access Knockout.js Bound Data #

Since the content is bound via Knockout’s html binding, you can execute JavaScript to fetch the raw bound data directly:

# Get the raw resultsContent value from Knockout's binding
results_content = driver.execute_script("""
    const element = document.getElementById('divresults');
    return ko.dataFor(element).resultsContent;
""")
print(results_content)

# Or get the fully rendered HTML directly via JS
rendered_html = driver.execute_script("return document.getElementById('divresults').innerHTML;")
print(rendered_html)

These methods should help you capture the full content that’s visible in your browser’s dev tools.

内容的提问来源于stack exchange，提问作者user9794893