使用Selenium提取Copilot搜索结果超时求助：无法定位元素

• 已成功定位Copilot搜索栏并执行关键词搜索，但无法定位页面上的搜索结果
• 需要将提取的搜索结果导入Excel表格
• 使用工具：Selenium、OpenPyxl、Pandas、Chrome WebDriver

#Function to retrieve the result text with retries
def get_result_text(driver, company_name, retries=3, wait_time=30):
    for attempt in range(retries):

        try:
            #Wait for the result container to load
            result_host = WebDriverWait(driver, 20).until(
                EC.presence_of_element_located((By.CSS_SELECTOR, "cib-chat-main"))
            ).shadow_root
            print(f"Shadow result host found for {company_name}")

            shadow_root_result1 = get_shadow_root(driver, result_host)
            result_turn_host = shadow_root_result1.find_element(By.CSS_SELECTOR, "cib-chat-turn")
            shadow_root_result2 = get_shadow_root(driver, result_turn_host)
            message_group_host = shadow_root_result2.find_element(By.CSS_SELECTOR, "cib-message-group")
            shadow_root_result3 = get_shadow_root(driver, message_group_host)
            message_host = shadow_root_result3.find_element(By.CSS_SELECTOR, "cib-message")
            shadow_root_result4 = get_shadow_root(driver, message_host)
            shared_host = shadow_root_result4.find_element(By.CSS_SELECTOR, "cib-shared")
            shadow_root_result5 = get_shadow_root(driver, shared_host)
            result_text_element = shadow_root_result5.find_element(By.CSS_SELECTOR, "div")
            print(f"Result text located for {company_name}")

            # Extract the text content from the results
            return result_text_element.text

DevTools listening on ws://127.0.0.1:63878/devtools/browser/2764cdc1-f139-425f-9774-b7ed1788f4bf
Columns in the DataFrame: Index(['Company'], dtype='object')
Shadow root 1 found for Plastic Omnium
Shadow roots 2 and 3 found for Plastic Omnium
Search initiated for Plastic Omnium
Created TensorFlow Lite XNNPACK delegate for CPU.
Timeout error processing Plastic Omnium, attempt 1 of 3
Timeout error processing Plastic Omnium, attempt 2 of 3
Timeout error processing Plastic Omnium, attempt 3 of 3
Timeout error processing Plastic Omnium: Message: 
Stacktrace:
        GetHandleVerifier [0x00007FF766E8EEA2+31554]
        (No symbol) [0x00007FF766E07ED9]
        (No symbol) [0x00007FF766CC872A]
        (No symbol) [0x00007FF766D18434]
        (No symbol) [0x00007FF766D1853C]
        (No symbol) [0x00007FF766D5F6A7]
        (No symbol) [0x00007FF766D3D06F]
        (No symbol) [0x00007FF766D5C977]
        (No symbol) [0x00007FF766D3CDD3]
        (No symbol) [0x00007FF766D0A33B]
        (No symbol) [0x00007FF766D0AED1]
        GetHandleVerifier [0x00007FF767198B1D+3217341]
        GetHandleVerifier [0x00007FF7671E5AE3+3532675]
        GetHandleVerifier [0x00007FF7671DB0E0+3489152]
        GetHandleVerifier [0x00007FF766F3E776+750614]
        (No symbol) [0x00007FF766E1375F]
        (No symbol) [0x00007FF766E0EB14]
        (No symbol) [0x00007FF766E0ECA2]
        (No symbol) [0x00007FF766DFE16F]
        BaseThreadInitThunk [0x00007FF8940E7344+20]
        RtlUserThreadStart [0x00007FF89595CC91+33]

问题分析 #

Copilot页面采用多层Shadow DOM结构，原代码存在以下问题：

• 链式获取Shadow DOM和元素，未对每一步元素做显式等待，容易超时
• CSS选择器不精准，比如cib-chat-turn未区分用户和AI的消息，导致定位错误
• 使用presence_of_element_located仅确认元素存在，未确保元素可见，可能元素未加载完成

修改后的代码实现 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

def get_shadow_root(element):
    return element.shadow_root

def get_result_text(driver, company_name, retries=3):
    for attempt in range(retries):
        try:
            # 等待cib-chat-main元素加载并获取其shadow root
            cib_chat_main = WebDriverWait(driver, 30).until(
                EC.visibility_of_element_located((By.CSS_SELECTOR, "cib-chat-main"))
            )
            shadow_root_main = get_shadow_root(cib_chat_main)
            
            # 等待AI的聊天消息容器加载（区分author="ai"）
            cib_chat_turn = WebDriverWait(shadow_root_main, 30).until(
                EC.visibility_of_element_located((By.CSS_SELECTOR, 'cib-chat-turn[author="ai"]'))
            )
            shadow_root_turn = get_shadow_root(cib_chat_turn)
            
            # 等待消息组加载
            cib_message_group = WebDriverWait(shadow_root_turn, 30).until(
                EC.visibility_of_element_located((By.CSS_SELECTOR, "cib-message-group"))
            )
            shadow_root_group = get_shadow_root(cib_message_group)
            
            # 等待消息元素加载
            cib_message = WebDriverWait(shadow_root_group, 30).until(
                EC.visibility_of_element_located((By.CSS_SELECTOR, "cib-message"))
            )
            shadow_root_message = get_shadow_root(cib_message)
            
            # 等待共享内容容器加载
            cib_shared = WebDriverWait(shadow_root_message, 30).until(
                EC.visibility_of_element_located((By.CSS_SELECTOR, "cib-shared"))
            )
            shadow_root_shared = get_shadow_root(cib_shared)
            
            # 获取结果文本（选择所有div文本拼接，避免遗漏）
            result_divs = shadow_root_shared.find_elements(By.CSS_SELECTOR, "div")
            result_text = "\n".join([div.text for div in result_divs if div.text.strip()])
            
            print(f"成功提取{company_name}的搜索结果")
            return result_text
        
        except Exception as e:
            print(f"处理{company_name}时出错，第{attempt+1}次重试：{str(e)}")
            # 重试前刷新页面并重新发起搜索
            driver.refresh()
            # 此处需调用你的搜索函数重新执行搜索，示例：search_company(driver, company_name)
    return None

# 提取结果并保存到Excel
def save_to_excel(results, filename="copilot_results.xlsx"):
    df = pd.DataFrame(results, columns=["Company", "Result"])
    df.to_excel(filename, index=False)
    print(f"结果已保存到{filename}")

# 使用示例
if __name__ == "__main__":
    driver = webdriver.Chrome()
    driver.get("https://copilot.microsoft.com/")
    
    # 假设已完成登录和搜索动作，以单个公司为例
    company = "Plastic Omnium"
    # 调用你的搜索函数执行搜索，示例：search_company(driver, company)
    result = get_result_text(driver, company)
    
    if result:
        save_to_excel([[company, result]])
    
    driver.quit()

关键优化点 #

1. 每一步显式等待可见性：使用visibility_of_element_located替代presence_of_element_located，确保元素已加载完成且可见
2. 精准选择AI消息：通过cib-chat-turn[author="ai"]定位AI生成的结果，避免定位到用户输入的消息
3. 拼接所有div文本：搜索结果可能分散在多个div中，拼接所有非空文本确保提取完整内容
4. 重试时刷新页面：每次失败后刷新页面并重新发起搜索，避免页面状态异常导致的超时
5. 完善异常捕获：捕获所有异常并输出具体错误信息，便于调试

内容的提问来源于stack exchange，提问作者dhruvesh gharda