Webscraping #shadow-root(open)数据爬取失败求助：代码异常排查

问题概述 #

尝试爬取页面中#shadow-root(open)内的展商数据，现有代码执行后输出全为"Couldn't Find"，期望获取以下结果：

Name: ACEDS - Association of Certified E-Discovery Specialists
Booth: 828
Url: https://www.aceds.org/

错误原因分析 #

1. 方法误用：使用find_elements()返回元素列表，列表对象没有.text和.get_attribute()方法，应使用find_element()（单数形式）获取单个元素
2. 选择器定位错误：官网链接的选择器指向了div.exhibitor__meta，但实际href属性在该div下的<a>标签中
3. 等待条件不足：仅等待元素存在（presence_of_element_located），未等待元素可见，可能导致元素未渲染完成就进行操作

修正后的代码 #

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time as t

options = webdriver.ChromeOptions()
options.add_argument("--no-sandbox")
options.add_argument('disable-notifications')
options.add_argument("start-maximized")
options.add_experimental_option("detach", True)
browser = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)

url = "https://iltacon2022.expofp.com/?aceds-association-of-certified-e-discovery-specialists"
browser.get(url)

# 等待宿主元素加载并获取shadow root
exhibitor_el = WebDriverWait(browser, 15).until(
    EC.visibility_of_element_located((By.XPATH, '//div[@data-event-id="iltacon2022"]/div'))
)
exhibitor_el_shadow_root = exhibitor_el.shadow_root

# 等待shadow dom内的展商信息加载完成
WebDriverWait(exhibitor_el_shadow_root, 15).until(
    EC.visibility_of_element_located((By.CSS_SELECTOR, 'div.exhibitor__bar'))
)

# 获取展商名称
try:
    name = exhibitor_el_shadow_root.find_element(By.CSS_SELECTOR, "div.exhibitor__bar").text
except:
    name = "Couldn't Find"

# 获取展位号
try:
    booth = exhibitor_el_shadow_root.find_element(By.CSS_SELECTOR, "a.exhibitor__categories-booth").text
except:
    booth = "Couldn't Find"

# 获取官网链接
try:
    url = exhibitor_el_shadow_root.find_element(By.CSS_SELECTOR, "div.exhibitor__meta a").get_attribute('href')
except:
    url = "Couldn't Find"

print(f"Name: {name}")
print(f"Booth: {booth}")
print(f"Url: {url}")
browser.quit()

验证结果 #

执行修正后的代码，输出如下：

Name: ACEDS - Association of Certified E-Discovery Specialists
Booth: 828
Url: https://www.aceds.org/

内容的提问来源于stack exchange，提问作者BQuist