Selenium无法识别<c-practitioner-search>等自定义标签的问题咨询

尝试用Python Selenium抓取动态网站数据，目标站点包含自定义Web组件标签如<c-practitioner-search>，以及带c-practitionersearch_practitionersearch属性的div标签，遇到以下问题：

• 将自定义标签视为普通标签调用find_elements()无法识别
• 遍历body下元素仅能获取约20%页面元素，但浏览器开发者工具可查看完整元素
• 静态解析工具（如BeautifulSoup）因网站动态渲染无法生效

HTML片段 #

<div data-item-id="256b3a7e-2963-49b0-9841-1397b03c7f02">
  <div data-priority class="ui-widget"> 
    <c-practitioner-search data-data-rendering-service-uid="40">
      <div c-practitionersearch_practitionersearch="">
        <div c-practitionersearch_practitionersearch="">
          <lightning-layout c-practitionersearch_practitionersearch="" class="bread-crumbs-style slds-grid">
            <slot lwc-4p3ig5mhdla class="slds-slot">
              <lightning-layout-item c-practitionersearch_practitionersearch>
                <slot>
                  <a c-practitionersearch_practitionersearch class="search-result-name-text-style">  Title </a>
                  <small c-practitionersearch_practitionersearch>
                     <ul c-practitionersearch_practitionersearch class="browser-default">
                       <li c-practitionersearch_practitionersearch class="search-result-text-style"> Info 1 </li>
                       <li c-practitionersearch_practitionersearch class="search-result-text-style"> Info 2 </li>
                       <li c-practitionersearch_practitionersearch class="search-result-text-style"> Info 3 </li>
                     </ul>
                   </small>
                 </slot>
               </lightning-layout-item>
             </slot>
           </lightning-layout>
         </div>
       </div>
     </c-practitioner-search>
   </div>
 </div>

Selenium代码片段 #

class PractitionerScraper:
    def __init__(self, url:str):
        main_path = pathlib.Path().absolute()

        driver_path = main_path / "chromedriver.exe"
        service = Service(executable_path= driver_path)

        chrome_options = webdriver.ChromeOptions()
        chrome_options.add_experimental_option('excludeSwitches', ['enable-logging'])

        self.driver = webdriver.Chrome(options=chrome_options, service=service)
        self.driver.implicitly_wait(4)
        self.driver.get(url)

    def search(self, time_to_load:int):
        time.sleep(time_to_load)
        main_div = self.driver.find_element(By.CSS_SELECTOR, "body div[class*='main']")

        all_tags = main_div.find_elements(By.CSS_SELECTOR, "*")
        for tag in all_tags:
            print(tag.tag_name)

1. 替换固定等待为显式等待 #

time.sleep()无法保证元素完全渲染，改用Selenium显式等待针对目标元素设置加载条件：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def search(self):
    # 等待自定义组件加载完成，超时时间10秒
    WebDriverWait(self.driver, 10).until(
        EC.presence_of_element_located((By.TAG_NAME, "c-practitioner-search"))
    )
    # 后续元素定位操作

2. 正确定位自定义组件与属性元素 #

自定义标签可直接通过标签名或CSS选择器定位，带特殊属性的元素用属性选择器：

# 定位所有<c-practitioner-search>标签
practitioner_components = self.driver.find_elements(By.TAG_NAME, "c-practitioner-search")

# 定位带c-practitionersearch_practitionersearch属性的div
target_divs = self.driver.find_elements(By.CSS_SELECTOR, "div[c-practitionersearch_practitionersearch]")

3. 处理Shadow DOM（核心解决方法） #

自定义Web组件通常用Shadow DOM封装内部元素，Selenium默认无法直接访问，需通过JavaScript获取Shadow Root后操作：

# 获取自定义组件的Shadow Root
script = """
return document.querySelector('c-practitioner-search').shadowRoot;
"""
shadow_root = self.driver.execute_script(script)

# 在Shadow Root内提取数据
title = shadow_root.find_element(By.CSS_SELECTOR, "a.search-result-name-text-style").text.strip()
info_list = [item.text.strip() for item in shadow_root.find_elements(By.CSS_SELECTOR, "li.search-result-text-style")]

print(title)
print(info_list)

4. 递归遍历包含Shadow DOM的所有元素 #

如果需要遍历页面全部元素，需递归处理Shadow Root内的内容：

def get_all_elements(self, element):
    elements = []
    # 获取当前元素的子元素
    children = element.find_elements(By.CSS_SELECTOR, "*")
    for child in children:
        elements.append(child)
        # 检查并处理Shadow Root
        shadow_root = self.driver.execute_script("return arguments[0].shadowRoot;", child)
        if shadow_root:
            elements.extend(self.get_all_elements(shadow_root))
    return elements

# 使用示例
main_div = self.driver.find_element(By.CSS_SELECTOR, "body div[class*='main']")
all_elements = self.get_all_elements(main_div)
for elem in all_elements:
    print(elem.tag_name)

5. 检查并切换iframe #

若目标元素嵌套在iframe内，需先切换到iframe再操作：

# 定位iframe并切换上下文
iframe = self.driver.find_element(By.CSS_SELECTOR, "iframe")
self.driver.switch_to.frame(iframe)

# 执行元素定位与数据提取...

# 操作完成后切回主文档
self.driver.switch_to.default_content()

内容的提问来源于stack exchange，提问作者Far1D