无法从网站提取Bed字段Span数据的Selenium技术求助

你遇到的核心问题是元素定位不准确或等待机制不可靠，导致无法提取Beds数据，以下是具体修复方案：

1. 替换固定等待为显式等待 #

time.sleep(10)是固定时长等待，无法适配页面实际渲染速度，改用Selenium显式等待，直到目标元素可见或可交互，稳定性更强。

2. 修正元素定位逻辑 #

原CSS选择器可能不符合页面实际结构，尝试两种更通用的定位方式：

• 优先通过data-qa属性直接定位数值元素（多数房产网站会把标签和数值拆分到不同元素，比如标签是span[data-qa="beds"]，数值是span[data-qa="beds-value"]）
• 如果上述方式失效，通过文本定位Beds标签，再取其相邻的数值元素

修改后的完整代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def extract_beds_from_url(url):
    beds_data = {'URL': url, 'Beds': 'N/A'}
    try:
        # 初始化WebDriver
        print("Setting up the WebDriver...")
        service = Service(ChromeDriverManager().install())
        driver = webdriver.Chrome(service=service)
        print("WebDriver set up successfully.")

        # 打开目标页面
        print(f"Opening URL: {url}")
        driver.get(url)
        print("URL opened successfully.")

        # 显式等待，最长20秒直到元素加载完成
        wait = WebDriverWait(driver, 20)
        
        # 尝试第一种定位方式：直接通过data-qa属性找数值元素
        try:
            print("Attempting to find beds value via data-qa attribute...")
            beds_element = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'span[data-qa="beds-value"]')))
            beds_data['Beds'] = beds_element.text.strip()
            print(f"Extracted Beds: {beds_data['Beds']}")
        except Exception as e:
            print(f"Failed with data-qa selector: {e}")
            # 尝试第二种定位方式：通过标签找相邻数值元素
            try:
                print("Attempting to find beds via sibling element...")
                beds_label = wait.until(EC.visibility_of_element_located((By.XPATH, '//span[contains(text(), "Beds")]')))
                beds_element = beds_label.find_element(By.XPATH, './following-sibling::span')
                beds_data['Beds'] = beds_element.text.strip()
                print(f"Extracted Beds: {beds_data['Beds']}")
            except Exception as e2:
                print(f"Failed with sibling selector: {e2}")

        # 关闭WebDriver
        print("Closing WebDriver...")
        driver.quit()
        print("WebDriver closed successfully.")

    except Exception as e:
        print(f"An error occurred during execution: {e}")

    return beds_data

# 目标页面URL
page_url = 'https://portal.onehome.com/es-US/property/aotf~1075849001~SEFMIAMI?token=eyJPU04iOiJTRUZNSUFNSSIsInR5cGUiOiIxIiwiY29udGFjdGlkIjo5MjgxMjg5LCJzZXRrZXkiOiI3MzAiLCJlbWFpbCI6InRhYmlvMTk4NUBob3RtYWlsLmNvbSIsInJlc291cmNlaWQiOjAsImFnZW50aWQiOjEwOTc1LCJpc2RlbHRhIjpmYWxzZSwiVmlld01vZGUiOiIxIn0%3D&searchId=d198393a-22a6-33b5-a207-8db32bb8cf89'

# 提取数据并输出
beds_data = extract_beds_from_url(page_url)
print("Extracted Data:")
print(f"URL: {beds_data['URL']}")
print(f"Beds: {beds_data['Beds']}")

额外排查建议 #

• 手动打开页面，右键检查Beds数值对应的元素，确认其data-qa属性、类名或相邻元素结构，按需调整定位器
• 如果页面需滚动才显示目标内容，可添加滚动操作：driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

内容的提问来源于stack exchange，提问作者tabio 85