Selenium TAG_NAME定位Shadow DOM内元素失败问题求助

问题描述 #

• 对目标网站做数据爬取时需要定位指定元素，目标元素的name、id属性均为动态生成，无法使用XPATH、CSS_SELECTOR、ID三种定位方式，尝试通过标签名定位元素执行失败
• 定位逻辑为先找到父级li元素，再调用find_element()方法查找其下第一个子元素<results-card>始终无法成功，必须定位到该元素的原因是这个元素挂载了需要展开的Shadow Root
• 排查时发现页面原始源代码展示的HTML结构，和浏览器开发者工具Elements面板中展示的结构存在差异，不清楚问题原因和解决方法

现有代码 #

nb_cv=0
liens_profils=[]
nb_cv_dmd=int(nb_cv_dmd)
while nb_cv<nb_cv_dmd:

  #on ouvre le premier shadow root

    WebDriverWait(driver,20).until(EC.presence_of_element_located((By.XPATH,'//*[@id="resultslist"]')))
    liste_profils_shadow= expand_shadow_element(driver.find_element(by=By.XPATH,value='//*[@id="resultslist"]'))
    #on recupere l'élément parent des profils et on récupère tous les profils
    parent_profils=liste_profils_shadow.find_element(by=By.ID,value="list")
    profils=parent_profils.find_elements(by=By.TAG_NAME,value="li")
    for profil in profils:
        if nb_cv<nb_cv_dmd:
            try:
                #WebDriverWait(driver,20).until(EC.presence_of_element_located((By.TAG_NAME,"results-card")))
                result_card=profil.find_element(by=By.TAG_NAME,value="results-card")
                result_card_shadow=expand_shadow_element(result_card)

待定位元素参考 #

对应HTML结构 #

<!-- Liste de résultats -->
        <results-list class="resultslist"
                      id="resultslist"
                      state=[{"idCandidat":17595816,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":14747406,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":17816114,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":15634996,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":17641294,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":17808000,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":16696140,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":13730612,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":16623996,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":17435668,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":17809379,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":15037379,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":17781323,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":10005222,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":17805944,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":17415932,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":13931906,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":15098652,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":17745872,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}},{"idCandidat":17790259,"isSelected":false,"iscollapsed":true,"hello":{"helloState":"","helloNeedFeedback":false}}] withPreview>
                    <template slot="17595816" data-results-card>
                        


<results-card id="person_17595816"
              data-candidat-id="17595816"
              data-over-url="/Recherche/CvOver?candidateId=17595816&amp;guid=4739f03b-bc68-47e5-ba5c-9a79f1978d44&amp;Position=1&amp;NbResultats=966&amp;isAnonymized=False&amp;IdSource=1&amp;OngletProjet=Candidats&amp;OngletSelectionne=ParDate&amp;Sources=1"
              mobility='{
        "otherDeps":["95","78","91","94","77","92","75","93"],
        "homeDep":"76",
        "isForeign":false,
        "location":"Saint-&#201;tienne-du-Rouvray",
        "mobility":[]
    }'
              infos='{
        "qualification":"3",
        "exp":"1",
        "posteRecherche":"D&#233;veloppeur web, \u003cspan class=\"highlight\"\u003eDevOps\u003c/span\u003e, D&#233;veloppeur Java JEE",
        "sector":["Informatique / T&#233;l&#233;com / Multim&#233;dia"],
        "salary":"&#192; convenir",
        "handicap":false,
        "dispo":"Imm&#233;diate"
    }'
              data='{
        "posteActuel":"Ing&#233;nieur d\u0027&#233;tudes et d&#233;veloppement informatique",
        "name":"Ulrich KENMOGNE KAMGUIA",
        "linkUrl": "/Profil?CandidateId=17595816&amp;Position=1&amp;NbResultats=966&amp;NbParPages=11&amp;IdSource=1&amp;Sources=System.Collections.Generic.List%601%5BSystem.Int32%5D&amp;GuidRecherche=4739f03b-bc68-47e5-ba5c-9a79f1978d44&amp;OngletSelectionne=ParDate&amp;CvCurrentPage=1&amp;NbPageCv=0&amp;OngletProjet=Candidats&amp;FromSelection=False&amp;FromNotification=False&amp;StartFrom=0&amp;IndexCandidatListe=0&amp;CurrentPosition=0&amp;IdRecherche=0&amp;SourceOriginePage=Recherche&amp;sources=1",
        "sitePicto":"BDM/job",
        "sitePictoLabel":"BDM/job",
        "date":"21/06/2022 13:08:05",
        "isMoreThan12Month":false,
        "idCandidat":"17595816"
    }'
              buttonsData='{
        "idCandidat":"17595816",
        "idProjet":"",
        "name":"Ulrich KENMOGNE KAMGUIA",
        "idSource":"1",
        "currentPosition":"1",
        "currentCandidate":"17595816",
        "sources":"1",
        "extensionOriginale":"",
        "hash":"b625f3af37973803d90b964f04da93d3"
    }'
              tags='[]'>

运行报错信息 #

Message: invalid argument: invalid locator
  (Session info: chrome=102.0.5005.115)
Stacktrace:
Backtrace:
        Ordinal0 [0x0113D953+2414931]
        Ordinal0 [0x010CF5E1+1963489]
        Ordinal0 [0x00FBC6B8+837304]
        Ordinal0 [0x00FE9691+1021585]
        Ordinal0 [0x00FE97EB+1021931]
        Ordinal0 [0x00FDFD21+982305]
        Ordinal0 [0x010044E4+1131748]
        Ordinal0 [0x00FDFC74+982132]
        Ordinal0 [0x010046B4+1132212]
        Ordinal0 [0x01014812+1198098]
        Ordinal0 [0x010042B6+1131190]
        Ordinal0 [0x00FDE860+976992]
        Ordinal0 [0x00FDF756+980822]
        GetHandleVerifier [0x013ACC62+2510274]
        GetHandleVerifier [0x0139F760+2455744]
        GetHandleVerifier [0x011CEABA+551962]
        GetHandleVerifier [0x011CD916+547446]
        Ordinal0 [0x010D5F3B+1990459]
        Ordinal0 [0x010DA898+2009240]
        Ordinal0 [0x010DA985+2009477]
        Ordinal0 [0x010E3AD1+2046673]
        BaseThreadInitThunk [0x76C9FA29+25]
        RtlGetAppContainerNamedObjectPath [0x777A7A7E+286]
        RtlGetAppContainerNamedObjectPath [0x777A7A4E+238]

问题原因 #

1. 源码和Elements面板结构差异是正常现象：页面原始源代码是服务端首次返回的静态HTML，其中<results-card>包裹在<template>标签内，这类标签的内容是惰性的，不会直接渲染到DOM树；开发者工具Elements面板展示的是前端JS执行完成、所有Web Component实例化、Shadow DOM挂载完成后的实时DOM结构。
2. 定位器报错核心原因：Selenium 4.0以前的版本，在Shadow Root节点下使用By.TAG_NAME定位带连字符的自定义Web Component标签时存在兼容问题，会触发无效定位器报错；且代码执行时如果<results-card>还没从template标签中渲染插入到li节点下，也会查找失败。
3. 原代码存在缩进错误，profils=parent_profils.find_elements(...)行缩进不规范，会触发语法异常。

修复方案 #

1. 替换定位方式：不要用By.TAG_NAME定位自定义元素，改用CSS选择器results-card做定位，兼容性更好。
2. 加显式等待：遍历每个li节点后，等待内部的results-card元素渲染完成再做后续操作，避免元素还没实例化就查找。
3. 修正代码缩进问题，统一使用4空格缩进。

修正后的核心代码片段：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

def expand_shadow_element(element):
    # 标准Shadow Root展开实现
    return driver.execute_script("return arguments[0].shadowRoot", element)

nb_cv=0
liens_profils=[]
nb_cv_dmd=int(nb_cv_dmd)
while nb_cv<nb_cv_dmd:
    # 等待外层resultslist加载并展开第一层shadow root
    results_list = WebDriverWait(driver,20).until(
        EC.presence_of_element_located((By.XPATH,'//*[@id="resultslist"]'))
    )
    liste_profils_shadow = expand_shadow_element(results_list)
    # 获取列表父节点
    parent_profils = liste_profils_shadow.find_element(By.ID, "list")
    # 等待所有li节点加载
    profils = WebDriverWait(liste_profils_shadow, 20).until(
        EC.presence_of_all_elements_located((By.TAG_NAME, "li"))
    )
    for profil in profils:
        if nb_cv >= nb_cv_dmd:
            break
        try:
            # 等待li下的results-card渲染完成，用CSS选择器定位替代TAG_NAME
            result_card = WebDriverWait(profil, 10).until(
                EC.presence_of_element_located((By.CSS_SELECTOR, "results-card"))
            )
            result_card_shadow = expand_shadow_element(result_card)
            # 后续爬取逻辑
            nb_cv +=1
        except Exception as e:
            # 自定义异常处理逻辑
            continue

内容的提问来源于stack exchange，提问作者Our1