如何用Python Selenium WebDriver一键展开任意网站的所有隐藏字段

需求：一键展开目标网页中的所有隐藏字段，现有代码仅能针对特定网站逐个操作（示例代码如下），需要实现适配任意网页的通用方案。

示例特定网站代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

service = Service(executable_path='chromedriver.exe')
options = webdriver.ChromeOptions()
driver = webdriver.Chrome(service=service, options=options)

driver.get('https://kpmg.com/xx/en/home/about/offices.html')

driver.find_element(By.XPATH, "//*[@id='country24-accordion']").click()

示例隐藏字段效果 #

核心思路 #

网页中的隐藏/折叠内容通常有统一特征，通用方案需基于这些特征批量定位并操作：

• 元素class包含accordion、collapse、toggle、expand等关键词
• 带有aria-expanded="false"属性（表示当前处于折叠状态）
• 文本内容为“展开”“更多”“Expand”“More”等提示性文字
• 点击后触发内容显示的按钮/标题元素

通用代码实现 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import ElementClickInterceptedException, StaleElementReferenceException

def expand_all_hidden_elements(driver):
    # 定义多维度定位规则，覆盖绝大多数折叠场景
    locators = [
        (By.CSS_SELECTOR, '[aria-expanded="false"]'),
        (By.CSS_SELECTOR, '.accordion:not([aria-expanded="true"])'),
        (By.CSS_SELECTOR, '.collapse-toggle, .expand-btn'),
        (By.XPATH, '//button[contains(text(),"展开") or contains(text(),"更多")]'),
        (By.XPATH, '//button[contains(text(),"Expand") or contains(text(),"More")]'),
        (By.TAG_NAME, 'summary')  # HTML原生折叠组件触发元素
    ]

    wait = WebDriverWait(driver, 10)
    
    for by, selector in locators:
        try:
            elements = wait.until(EC.presence_of_all_elements_located((by, selector)))
            for elem in elements:
                try:
                    # 跳过已展开元素
                    if elem.get_attribute('aria-expanded') == 'true':
                        continue
                    # 滚动到元素可见位置，避免点击失败
                    driver.execute_script("arguments[0].scrollIntoView(true);", elem)
                    elem.click()
                except (ElementClickInterceptedException, StaleElementReferenceException):
                    # 处理元素被遮挡、已失效的情况，直接跳过
                    continue
        except:
            # 当前定位规则无匹配元素，跳过继续下一个规则
            continue

# 初始化浏览器
options = webdriver.ChromeOptions()
# 可选：启用无头模式（后台运行）
# options.add_argument('--headless=new')
service = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=service, options=options)

# 替换为你的目标网页URL
driver.get("https://example.com/target-page")

# 执行一键展开
expand_all_hidden_elements(driver)

# 后续可添加内容提取、保存等操作
# ...

# 关闭浏览器
# driver.quit()

自定义适配技巧 #

如果目标网站有特殊的折叠元素设计，可在locators列表中新增定位规则：

• 针对特定ID前缀：(By.CSS_SELECTOR, '[id^="collapse-"]')
• 针对自定义属性：(By.CSS_SELECTOR, '[data-action="expand"]')
• 针对特定文本的非按钮元素：(By.XPATH, '//div[contains(text(),"显示全部")]')

内容的提问来源于stack exchange，提问作者Goutam