Selenium点击动态楼层元素后页面未加载房源信息的问题

我是web scraping新手，尝试从动态网站抓取公寓房源信息，已成功获取默认19层的数据，但点击已知有房源的楼层元素后，页面始终显示「There are no available apartments on this floor」，无法获取对应楼层的房源信息，怀疑点击操作未触发正确的动态加载。

当前代码 #

xpath = '//*[@id="mobile-floor-carousel-list"]/a[11]'
# 点击楼层
floor = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, xpath))).click()

print("clicked on page")

floorplans = WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.XPATH, "//div[@id='unit-list-items']"))
)
print("waited for page to load")

print(driver.find_element(By.XPATH,"//html").get_attribute('innerHTML'))

目标楼层元素HTML #

<a href="#" class="skylease__toolbar-floor-link swiper-slide skylease__mobile-floor-link--has-units" data-js-hook="mobile-floor-selector" data-floor="14" style="width: 50.6154px; margin-right: 5px;"><div><span>Floor</span><span>14</span><span class="skylease-avail-count"><span data-floor="14" data-js-hook="available-unit-count">4</span> avail</span></div></a>

点击后返回的无房源innerHTML #

</div>
   <div id="unit-list" class="skylease__unit-list">
   <p id="unit-list-message" class="skylease__unit-message">There are no available apartments on this floor.</p>
   <div class="skylease__unit-list-items" id="unit-list-items"></div>
      </div>

手动操作预期的含房源innerHTML #

<div id="unit-list" class="skylease__unit-list">
                <p id="unit-list-message" class="skylease__unit-message" style="display: none;">There are no available apartments on this floor.</p>    
    <div class="skylease__unit-list-items" id="unit-list-items"><a href="#" class="skylease__unit-list-item skylease__unit-list-item--alt" data-unit="1504" style="display: block;">
        ....edited out for simplicity
        
                <div class="skylease__unit-list-item-info-wrap">
                    <div class="skylease__unit-list-item-info">
                        <div class="skylease__unit-list-item-details skylease__unit-list-item-details--unit">
                            #1504
        ...edited out for simplicity
                    <div class="skylease__unit-list-item-info">
                        <p class="skylease__unit-list-item-details">Studio</p>
                        <p class="skylease__unit-list-item-details">1 bath</p>
                        <p class="skylease__unit-list-item-details"></p>
                                            <p class="skylease__unit-list-item-details">512 sq. ft.</p>
                                        <p class="skylease__unit-list-item-details"></p>
                    </div>
        
                                    <div class="skylease__unit-list-item-info">
                            <p class="skylease__unit-list-item-details skylease__unit-list-item-details--price">
                                                        <span>1,807</span>
                                                            <span>3683</span>
                                                    </p>
                        </div>
                            </div>

1. 确保元素可交互后再点击 #

presence_of_element_located仅保证元素存在DOM中，但可能未完全渲染无法触发点击逻辑，改用element_to_be_clickable：

floor = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, xpath))).click()

2. 等待真实房源加载完成 #

当前等待的unit-list-items空元素也会满足条件，需等待房源元素出现或无房源提示被隐藏：

# 方案1：等待无房源提示隐藏（针对有房源的楼层）
WebDriverWait(driver, 10).until(
    EC.invisibility_of_element_located((By.ID, "unit-list-message"))
)

# 方案2：等待房源链接出现
floorplans = WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.XPATH, "//div[@id='unit-list-items']/a"))
)

3. 模拟真实用户点击行为 #

部分网站会检测非真实交互，可添加滚动或用ActionChains模拟点击：

# 先滚动到元素位置
floor_element = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, xpath))
)
driver.execute_script("arguments[0].scrollIntoView(true);", floor_element)
floor_element.click()

# 或者用ActionChains
from selenium.webdriver.common.action_chains import ActionChains
ActionChains(driver).move_to_element(floor_element).click().perform()

4. 跳过Selenium，直接请求API #

动态网站的房源数据通常来自后端API，可通过浏览器F12的Network面板，筛选XHR/fetch请求，找到加载楼层数据的接口（比如/api/floors/14），直接用requests库请求获取JSON数据，效率更高且避免交互问题。

内容的提问来源于stack exchange，提问作者user2634937