如何用Python的Selenium提取页面树中的href链接？

我尝试抓取目标页面嵌套在页面树内的「Field」链接，但目前只能获取一级链接，那些需要点击展开页面树才会显示的三级链接拿不到。推测是这些链接属于动态生成，之前试过用页面树内部元素的XPATH，也没得到结果。想请教是否需要添加点击展开页面树的代码？是否需要使用其他库？

当前代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")

driver = webdriver.Chrome(options=options)

url = 'https://ddwiki.reso.org/display/DDW17/'
driver.get(url)

elem = driver.find_element(By.CLASS_NAME, "plugin_pagetree_children_list")

links=elem.find_elements(By.TAG_NAME, "a")

for lnk in links:
   print(lnk.get_attribute('href'))

driver.quit()

现有输出（节选） #

https://ddwiki.reso.org/display/DDW17/Data+Dictionary+v1.7+Introduction?src=contextnavpagetreemode

https://ddwiki.reso.org/display/DDW17/Property+Resource?src=contextnavpagetreemode

https://ddwiki.reso.org/display/DDW17/Member+Resource?src=contextnavpagetreemode

https://ddwiki.reso.org/display/DDW17/Office+Resource?src=contextnavpagetreemode

https://ddwiki.reso.org/display/DDW17/Contacts+Resource?src=contextnavpagetreemode

相关HTML片段 #

<div class="plugin_pagetree_children" id="children393239-0">
    <ul class="plugin_pagetree_children_list" id="child_ul393239-0">
       <li>
          <div class="plugin_pagetree_childtoggle_container">
             <a id="plusminus393267-0" class="plugin_pagetree_childtoggle aui-icon aui-icon-small aui-iconfont-chevron-right" href="" data-type="toggle" data-page-id="393267" data-tree-id="0" aria-expanded="false" aria-label="Expand item Data Dictionary v1.7 Introduction">
             </a>
          </div>
          <div class="plugin_pagetree_children_content">
             <span class="plugin_pagetree_children_span" id="childrenspan393267-0">                        <a href="/display/DDW17/Data+Dictionary+v1.7+Introduction?src=contextnavpagetreemode">Data Dictionary v1.7 Introduction</a>
             </span>
          </div>
          <div id="children393267-0" class="plugin_pagetree_children_container">
          </div>
       </li>
       <li>
          <div class="plugin_pagetree_childtoggle_container">
             <a id="plusminus393372-0" class="plugin_pagetree_childtoggle aui-icon aui-icon-small aui-iconfont-chevron-down" href="" data-type="toggle" data-page-id="393372" data-tree-id="0" aria-expanded="true" aria-label="Expand item Property Resource" data-children-loaded="true" data-expanded="true">
             </a>
          </div>
          <div class="plugin_pagetree_children_content">
             <span class="plugin_pagetree_children_span" id="childrenspan393372-0">                        <a href="/display/DDW17/Property+Resource?src=contextnavpagetreemode">Property Resource</a>
             </span>
          </div>
          <div id="children393372-0" class="plugin_pagetree_children_container">
             <ul class="plugin_pagetree_children_list" id="child_ul393372-0">
                <li>
                   <div class="plugin_pagetree_childtoggle_container">
                      <a id="plusminus393402-0" class="plugin_pagetree_childtoggle aui-icon aui-icon-small aui-iconfont-chevron-down" href="" data-type="toggle" data-page-id="393402" data-tree-id="0" aria-expanded="true" aria-label="Expand item Business Group" data-children-loaded="true" data-expanded="true">
                      </a>
                   </div>
                   <div class="plugin_pagetree_children_content">
                      <span class="plugin_pagetree_children_span" id="childrenspan393402-0">                        <a href="/display/DDW17/Business+Group?src=contextnavpagetreemode">Business Group</a>
                      </span>
                   </div>
                   <div id="children393402-0" class="plugin_pagetree_children_container" style="display: block;">
                      <ul class="plugin_pagetree_children_list" id="child_ul393402-0">
                         <li>
                            <div class="plugin_pagetree_childtoggle_container">
                               <span class="no-children icon"></span>
                            </div>
                            <div class="plugin_pagetree_children_content">
                               <span class="plugin_pagetree_children_span" id="childrenspan393855-0">                        <a href="/display/DDW17/BusinessName+Field?src=contextnavpagetreemode">BusinessName Field</a>
                               </span>
                            </div>
                            <div id="children393855-0" class="plugin_pagetree_children_container">
                            </div>
                         </li>

你的推测完全正确：三级「Field」链接是动态加载的，必须先触发页面树的展开操作，浏览器才会请求并渲染这些深层内容。不需要更换库，用Selenium本身就能解决，核心逻辑是自动点击所有可展开的节点，再统一抓取目标链接。

修改后的完整代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
# 无头模式下设置窗口大小，避免元素因视口过小被隐藏
options.add_argument("window-size=1920,1080")

driver = webdriver.Chrome(options=options)
url = 'https://ddwiki.reso.org/display/DDW17/'
driver.get(url)

wait = WebDriverWait(driver, 10)

# 循环点击所有未展开的节点，直到没有可展开项为止
while True:
    # 定位所有箭头朝右的未展开按钮
    toggle_buttons = wait.until(EC.presence_of_all_elements_located(
        (By.CSS_SELECTOR, ".plugin_pagetree_childtoggle.aui-iconfont-chevron-right")
    ))
    if not toggle_buttons:
        break  # 无剩余可展开节点，退出循环
    for btn in toggle_buttons:
        try:
            # 用JS点击避免元素遮挡或不可交互的问题
            driver.execute_script("arguments[0].click();", btn)
            time.sleep(0.5)  # 给页面留加载时间
        except Exception:
            continue  # 跳过点击失败的节点

# 抓取所有包含「Field」关键词的链接
field_links = driver.find_elements(By.XPATH, "//a[contains(text(), 'Field')]")
for link in field_links:
    print(link.get_attribute('href'))

driver.quit()

关键细节说明 #

• 等待机制：用WebDriverWait确保元素加载完成，避免直接查找导致的空指针错误
• 节点展开逻辑：通过CSS选择器精准定位未展开的节点（箭头朝右），循环点击直到全部展开
• JS点击：相比直接调用btn.click()，JS点击能规避元素遮挡、不可交互等常见问题
• 精准筛选：最后用XPATH直接过滤包含「Field」的链接，无需再区分层级

内容的提问来源于stack exchange，提问作者infinitelylooped