使用Python Selenium抓取JS页面失败：无法定位元素报错求助

使用Python Selenium结合Chrome浏览器抓取https://data.cms.gov/tools/mapping-medicare-disparities-by-population页面，尝试定位siteContentWrapper元素时触发NoSuchElementException，代码及报错如下：

用户代码 #

import time 
import pandas as pd 
from selenium import webdriver 
from selenium.webdriver import Chrome 
from selenium.webdriver.chrome.service import Service 
from selenium.webdriver.common.by import By 
from webdriver_manager.chrome import ChromeDriverManager

options = webdriver.ChromeOptions() 
options.headless = True
options.page_load_strategy = 'none'
options.add_argument("--enable-javascript")
options.add_argument('--ignore-certificate-errors') 
chrome_path = ChromeDriverManager().install() 
chrome_service = Service(chrome_path) 

driver = Chrome(options=options, service=chrome_service) 

url = "https://data.cms.gov/tools/mapping-medicare-disparities-by-population" 

driver.get(url)
time.sleep(30)

siteContentWrapper1 = driver.find_element(By.CLASS_NAME, "siteContentWrapper")

报错信息 #

Traceback (most recent call last):

  File "CMS.py", line 30, in <module>
    siteContentWrapper1 = driver.find_element(By.CLASS_NAME, "siteContentWrapper")

  File "Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 1248, in find_element
    return self.execute(Command.FIND_ELEMENT, {

  File "Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 425, in execute
    self.error_handler.check_response(response)

  File "Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace)

NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":".siteContentWrapper"}
  (Session info: headless chrome=113.0.5672.127)

解决方案 #

1. 用显式等待替代固定sleep #

固定等待time.sleep(30)无法保证元素完全加载，改用WebDriverWait等待元素出现：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# 替换time.sleep(30)为：
wait = WebDriverWait(driver, 40)
siteContentWrapper1 = wait.until(EC.presence_of_element_located((By.CLASS_NAME, "siteContentWrapper")))

2. 检查目标元素是否在iframe内 #

访问页面后打开开发者工具，确认siteContentWrapper是否嵌套在iframe中。如果是，需先切换到iframe再定位元素：

# 示例：根据实际iframe的定位方式切换，此处以id为例
iframe = wait.until(EC.presence_of_element_located((By.ID, "target-iframe")))
driver.switch_to.frame(iframe)
# 定位目标元素
siteContentWrapper1 = wait.until(EC.presence_of_element_located((By.CLASS_NAME, "siteContentWrapper")))
# 操作完成后切回主文档
driver.switch_to.default_content()

3. 优化headless模式参数 #

Headless模式下Chrome可能因窗口尺寸问题导致元素渲染异常，添加窗口大小参数：

options.add_argument("--window-size=1920,1080")

4. 验证元素选择器准确性 #

确认页面中siteContentWrapper类名是否正确，或改用XPath定位提高稳定性：

siteContentWrapper1 = wait.until(EC.presence_of_element_located((By.XPATH, "//div[contains(@class, 'siteContentWrapper')]")))

修改后的完整代码示例 #

import pandas as pd 
from selenium import webdriver 
from selenium.webdriver import Chrome 
from selenium.webdriver.chrome.service import Service 
from selenium.webdriver.common.by import By 
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions() 
options.headless = True
options.page_load_strategy = 'none'
options.add_argument("--enable-javascript")
options.add_argument('--ignore-certificate-errors')
options.add_argument("--window-size=1920,1080")  # 添加窗口大小参数

chrome_path = ChromeDriverManager().install() 
chrome_service = Service(chrome_path) 

driver = Chrome(options=options, service=chrome_service) 

url = "https://data.cms.gov/tools/mapping-medicare-disparities-by-population" 

driver.get(url)

wait = WebDriverWait(driver, 40)

# 若页面存在iframe，需添加切换逻辑，此处假设无需切换
siteContentWrapper1 = wait.until(EC.presence_of_element_located((By.CLASS_NAME, "siteContentWrapper")))

# 后续操作...
driver.quit()

内容的提问来源于stack exchange，提问作者Abhishek Jain