使用BeautifulSoup提取嵌套Div数据失败求助：人口普查网站中位数收入

我需要从美国人口普查网站提取区域中位数收入数据，目标是搜索“Median income (dollars)”后的第一个结果，但页面元素的comp-id是动态生成的，导致原代码无法定位到目标元素。原代码尝试遍历嵌套div但无结果，代码如下：

import csv
import requests
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import pandas as pd
from bs4 import BeautifulSoup 


DRIVER_PATH = 'chromedriver_107.exe'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)


url = 'https://data.census.gov/cedsci/table?q=' + '53706' + '%20income&tid=ACSST5Y2020.S1901'
driver.get(url)

page = requests.get(url)


content = driver.page_source
soup = BeautifulSoup(content, 'lxml')

a = soup.findAll("div", {"comp-id":"1539"})
print(a)

问题分析 #

1. 动态元素定位失效：comp-id是动态生成的，每次页面加载都会变化，固定值无法匹配目标元素。
2. 页面加载未处理：直接获取driver.page_source可能赶在动态内容渲染完成前执行，导致获取的HTML不包含目标数据。
3. 冗余的requests请求：目标页面是JS动态渲染的，requests.get(url)无法获取到实际渲染后的内容，这一步完全多余。

解决方法 #

采用基于文本内容的稳定定位，结合Selenium的显式等待确保元素加载完成，完全避开动态变化的属性。

修改后的代码 #

import csv
import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

DRIVER_PATH = 'chromedriver_107.exe'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)

try:
    url = 'https://data.census.gov/cedsci/table?q=53706%20income&tid=ACSST5Y2020.S1901'
    driver.get(url)

    # 等待目标文本元素加载，最长等待10秒
    wait = WebDriverWait(driver, 10)
    median_income_label = wait.until(
        EC.presence_of_element_located((By.XPATH, "//*[contains(text(), 'Median income (dollars)')]"))
    )

    # 定位目标文本所在的行，提取对应数值单元格
    parent_row = median_income_label.find_element(By.XPATH, "./ancestor::div[contains(@class, 'tr')]")
    median_income_value = parent_row.find_element(By.XPATH, ".//div[contains(@class, 'td')][2]").text

    print(f"Median Income: {median_income_value}")

finally:
    driver.quit()

代码说明 #

1. 显式等待：用WebDriverWait等待目标文本元素出现，确保页面动态渲染完成后再操作。
2. 文本驱动定位：通过contains(text(), 'Median income (dollars)')定位目标行，不受动态ID影响。
3. 层级定位：先找到目标文本的父行，再提取该行的数值单元格，适配页面表格结构。
4. 资源清理：try...finally确保浏览器进程正常关闭，避免资源泄漏。

备选方案（结合BeautifulSoup） #

如果偏好使用BeautifulSoup解析，需确保获取完整渲染后的页面源码：

import pandas as pd
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

DRIVER_PATH = 'chromedriver_107.exe'
driver = webdriver.Chrome(executable_path=DRIVER_PATH)

try:
    url = 'https://data.census.gov/cedsci/table?q=53706%20income&tid=ACSST5Y2020.S1901'
    driver.get(url)

    # 等待表格容器加载完成
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, 'table-container'))
    )

    # 获取完整渲染后的页面源码
    content = driver.page_source
    soup = BeautifulSoup(content, 'lxml')

    # 通过文本匹配定位目标行，提取数值
    median_label = soup.find(text=lambda t: t and 'Median income (dollars)' in t)
    if median_label:
        parent_row = median_label.find_parent('div', class_='tr')
        if parent_row:
            median_value = parent_row.find('div', class_='td').find_next_sibling('div').text
            print(f"Median Income: {median_value}")

finally:
    driver.quit()

内容的提问来源于stack exchange，提问作者Zhe Wang