如何使用Python结合Selenium与Geckodriver提取指定网页文本？

需要提取的目标HTML片段：

<div class="jaHlC">
<div class="C" data-ft="true">
<div class="IuRIu">
<span>
<span class="biGQs _P fiohW uuBRH">
90 places sorted by traveler favorites</span>
</span>
<span class="nzZVd PJ">

用户尝试多种定位方式均失败，原代码如下：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = "https://www.tripadvisor.com/Attraction_Products-g28922-t21629-zfg21594-Alabama.html"

driver = webdriver.Firefox()
driver.get(url)

WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.ID, 'onetrust-accept-btn-handler'))).click()

# attempt 1 : does not work
#number = driver.find_element(By.XPATH, '//span[@class="biGQs _P fiohW uuBRH"]')

# attempt 2: does not work
#number = driver.find_element(By.XPATH, "/html/body/div[1]/main/div[1]/div/div[3]/div/div[2]/div[2]/div[2]/div/div/div[2]/div/div[2]/div/div/section[2]/div/div/div/span[1]/span")

# attempt 3: does not work either
number = driver.find_element(By.CSS_SELECTOR, "span.uuBRH")

1. 核心问题：未等待目标元素加载完成 #

点击cookie按钮后，页面主体内容仍在异步加载，直接查找元素会因元素未渲染完成而失败，必须显式等待目标元素出现。

2. 优化定位器 #

• 避免使用绝对XPATH（页面结构稍有变化就会失效）
• 优先使用稳定的相对定位或文本特征辅助定位

3. 完整可运行代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

url = "https://www.tripadvisor.com/Attraction_Products-g28922-t21629-zfg21594-Alabama.html"

driver = webdriver.Firefox()
driver.get(url)

# 接受cookie
WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.ID, 'onetrust-accept-btn-handler'))).click()

# 等待目标元素加载并提取文本
try:
    # 方式1：通过多类名组合定位（兼容类名顺序变化）
    target_element = WebDriverWait(driver, 20).until(
        EC.presence_of_element_located((By.XPATH, '//span[contains(@class, "biGQs") and contains(@class, "uuBRH")]'))
    )
    # 方式2：通过文本内容精准定位（推荐，不受类名变化影响）
    # target_element = WebDriverWait(driver, 20).until(
    #     EC.presence_of_element_located((By.XPATH, '//span[contains(text(), "places sorted by traveler favorites")]'))
    # )
    result_text = target_element.text.strip()
    print(result_text)
except Exception as e:
    print(f"提取失败：{str(e)}")
finally:
    driver.quit()

额外注意事项 #

• 若页面存在滚动加载，可提前滚动到元素位置：driver.execute_script("arguments[0].scrollIntoView();", target_element)
• 检查目标元素是否嵌套在iframe内，若有需先切换iframe：driver.switch_to.frame(iframe_element)

内容的提问来源于stack exchange，提问作者R Sandy