使用Selenium无法获取聊天机器人响应的技术求助

尝试通过Selenium实现与聊天机器人的自动化对话并存储其响应，但始终无法定位并获取响应内容。

聊天框HTML结构 #

.css-liwoqsn为响应元素类，.css-v9i7aw为提问元素类，展开.css-liwoqsn后，实际响应存储在<p>和<ul>标签中：

<div class="MuiBox-root css-v2kfba" data-section="copilot_chat">
    <div class="css-13г9wg5">
        <div class="MuiBox-root css-gajhq5"></div>
        <hr class="MuiDivider-root MuiDivider-fullWidth css-39bb06">
        <div class="css-1a9768g">
            <div class="css-liwoqsn">
                <div class="css-1gprod">
                    <div id="gtiriww2a0souw9" class="css-dcpOta">
                        <div class="MuiBox-root css-107u4gk">
                            <div class="content MuiBox-root css-x97jm9">
                                <div class="mb-2 leading-normal text-black prose prose-p:m-0 prose-a:font-medium prose-a:text-primary prose-a:no-underline hover:prose-a:underline prose-hr:my-3 prose-hr:border-0 prose-hr:border-t prose-hr:border-solid prose-hr:border-t-gray-300" style="font-size: 15px;">
                                    <p> </p>
                                    <ul></ul>
                                    <p> </p>
                                    <ul></ul>
                                </div>
                            </div>
                        </div>
                    </div>
                </div>
            </div>
            <div class="css-v9i7aw"></div>
            <div class="css-liwoqsn"></div>
            <div class="css-v9i7aw"></div>
            <div class="css-liwoqsn"></div>
            <div class="css-v9i7aw"></div>
            <div class="css-liwoqsn"></div>
            <div class="css-v9i7aw"></div>
            <div class="css-liwoqsn"></div>
        </div>
    </div>
</div>

响应元素定位补充信息 #

• CSS选择器：

#__next > main > div > span > div > div.MuiGrid-root.MuiGrid-item.MuiGrid-grid-md-5.css-114racd > div > div > div > div.css-1a9768q > div:nth-child(1)

• XPath（新响应始终位于顶部div）：

//*[@id="__next"]/main/div/span/div/div[2]/div/div/div/div[2]/div[1]

尝试的代码 #

chatbox = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, '.MuiInput-input.MuiInputBase-input.MuiInputBase-inputMultiline.css-zvroza')))

# Initiate conversation with the chatbot and ask each preset question
driver.implicitly_wait(15)
print ("Found textbox successfully")

# ==================== This is the section where i define how to question and retrieve answers from chatbox ============== #

# define function to send message to chatbox
def send_message(message):
    chatbox.send_keys(message)
    chatbox.send_keys(Keys.RETURN)
    print (message)
    time.sleep(10)

def get_response():
    print("in get response method")
    try:
        chat_messages = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '.css-liwoqsn')))
        latest_chat_message = chat_messages[0]

    except:
        latest_chat_message = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, '//*[@id="__next"]/main/div/span/div/div[2]/div/div/div/div[2]/div[1]')))

    # Get the response text
    response = latest_chat_message.find_element(By.CSS_SELECTOR, '.MuiBox-root.css-107u4gk').text
    print(response)
    # Add the response to the responses list
    responses.append(response)
    time.sleep(3) 
    return response

# ==================== This is the section where I question the chatbot using predefined questions ============== #

for question in questions.values():
        time.sleep(2)
        send_message(question)
        time.sleep(3)
        response = get_response()
        print ("=-=-=-=-=-=-=-=-=-=-=-=")
        responses.append(response)

问题排查与修复方案 #

1. 动态内容等待逻辑错误：
   固定time.sleep()无法适配响应加载的实际耗时，应使用显式等待直到响应内容可见，而非依赖休眠。同时避免隐式等待（driver.implicitly_wait()）和显式等待混用，两者逻辑冲突易导致定位异常。

2. 定位器稳定性不足：
   动态生成的CSS类名（如css-liwoqsn）可能随页面更新变化，建议结合静态属性（如data-section="copilot_chat"）和内容层级来定位，提升稳定性。

3. 文本获取层级错误：
   实际响应内容存储在.content容器下的<p>和<ul>标签中，直接获取上层容器的文本可能无法拿到完整内容，需精准定位到内容节点。

4. 异常处理逻辑缺陷：
   except块中使用presence_of_all_elements_located会返回元素列表，后续调用find_element会抛出类型错误，需统一元素定位的返回类型。

修改后的核心代码示例 #

# 移除隐式等待，统一使用显式等待
# driver.implicitly_wait(15)

def send_message(message):
    chatbox.clear()  # 清空输入框，避免重复输入
    chatbox.send_keys(message)
    chatbox.send_keys(Keys.RETURN)
    print(f"已发送: {message}")
    # 等待自己发送的消息出现，确认发送成功
    WebDriverWait(driver, 10).until(
        EC.visibility_of_element_located((By.CSS_SELECTOR, '[data-section="copilot_chat"] .css-v9i7aw:last-child'))
    )

def get_response():
    print("开始获取响应")
    # 等待最新响应的内容容器可见
    content_container = WebDriverWait(driver, 20).until(
        EC.visibility_of_element_located(
            (By.CSS_SELECTOR, '[data-section="copilot_chat"] .css-liwoqsn:first-child .content')
        )
    )
    # 提取所有p和ul标签的文本并拼接
    response_parts = content_container.find_elements(By.XPATH, './/p | .//ul')
    response_text = "\n".join([part.text.strip() for part in response_parts if part.text.strip()])
    print(f"响应内容:\n{response_text}")
    responses.append(response_text)
    return response_text

# 循环提问部分（移除重复的responses.append）
for question in questions.values():
    time.sleep(2)
    send_message(question)
    response = get_response()
    print("=-=-=-=-=-=-=-=-=-=-=-=")

内容的提问来源于stack exchange，提问作者Yahya Elghobashy