Python中使用Selenium无法在嵌套Shadow Root中定位元素

问题根源 #

1. 固定等待时长不足，weave-ordering元素或其Shadow Root未完全加载
2. 页面Shadow DOM层级判断错误，weave-ordering本身无Shadow Root
3. 元素选择器未匹配到正确的目标节点

修正方案 #

1. 替换固定等待为显式等待 #

用Selenium显式等待替代time.sleep，确保元素加载完成后再操作：

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome(DRIVER_PATH, options=options)
website = 'https://bodyandmind.com/california/longbeachdispensary/longbeachmenu/?category=flower#inventory'
driver.get(website)

# 等待年龄验证按钮并点击
wait = WebDriverWait(driver, 10)
age_btn = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'button.age-gate-submit.age-gate-submit-yes')))
age_btn.click()

2. 修正Shadow DOM层级 #

实际页面中weave-ordering无Shadow Root，正确的起始节点是weave-app，调整代码如下：

# 等待weave-app元素加载并获取其Shadow Root
weave_app = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "weave-app")))
root2 = weave_app.shadow_root

# 继续向下查找层级
root3 = root2.find_element(By.CSS_SELECTOR, "weave-inventory").shadow_root
root4 = root3.find_element(By.CSS_SELECTOR, "weave-product-list").shadow_root

# 获取所有产品节点
product_sections = root4.find_elements(By.CSS_SELECTOR, "weave-category-section weave-product")
for product in product_sections:
    product_root = product.shadow_root
    product_name = product_root.find_element(By.CSS_SELECTOR, ".product-container .product-name").text
    print(product_name)

3. 备选方案：直接调用API抓取数据 #

动态页面的产品数据通常来自API，打开浏览器开发者工具的Network标签，筛选XHR/Fetch请求，找到加载产品列表的接口，直接用requests调用更高效：

import requests

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
}
# 替换为实际抓取到的API地址
api_url = "https://api.bodyandmind.com/v1/inventory/locations/longbeach/products?category=flower"
response = requests.get(api_url, headers=headers)
products = response.json()

for item in products["data"]:
    print(item["name"])

注意事项 #

• 页面结构可能随网站更新变化，需定期通过开发者工具确认Shadow DOM层级
• 显式等待比固定等待更稳定，能避免因网络延迟导致的加载问题
• 优先选择API抓取，绕过复杂的Shadow DOM操作

内容的提问来源于stack exchange，提问作者Japneesh