使用Selenium 4.10.0滚动动态加载表格遇阻，求解决方案

问题详情 #

需要滚动某网站的动态加载表格，但常规滚动方法均失效：

• 使用send_keys(Keys.END)返回element not interactable错误
• 全局窗口滚动脚本driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")无效果
• ActionChains相关操作不起作用
• 尝试过多种元素定位方式（类、标签、XPATH、CSS_SELECTOR）均未解决

环境：Python 3.8、Selenium 4.10.0

尝试过的代码 #

from time import sleep
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as ES
from selenium.webdriver.support.ui import WebDriverWait as WDW
from selenium.webdriver.common.action_chains import ActionChains
from bs4 import BeautifulSoup as BS

self.URL_alnair = 'https://alnair.ae/app/view/1412/3386/apartment/apartments'
o = Options()
o.add_experimental_option('detach', True)
o.add_argument('--start-maximized')
chrome_driver_path = r"YOUR PATH"
driver = webdriver.Chrome(options=o,service=ChromeService(executable_path=chrome_driver_path))

def get_data():
    driver.get(self.URL_alnair)
    driver.set_page_load_timeout(2)
    
    element = driver.find_elements(By.CSS_SELECTOR, '._tr_18734_1')
    action.move_to_element(element[2])


def get_data():
    driver.get(self.URL_alnair)
    driver.set_page_load_timeout(2)
    
    driver.execute_script("window.scrollTo(0, 500)") # tried "document.body.scrollHeight", #"window.scrollBy(0, 200)", element.scrollTop = 500... etc


def get_data():
    driver.get(self.URL_alnair)
    driver.set_page_load_timeout(2)
    
    action = ActionChains(self.driver)

    element = self.driver.find_elements(By.CSS_SELECTOR, '._tr_18734_1')
    action.move_to_element(element[2]).perform()
    sleep(1)
    action.send_keys(Keys.ARROW_DOWN).perform() # 

解决方案 #

问题核心：表格内容在独立的滚动容器内，而非全局窗口，因此需要针对该容器执行滚动操作。

具体实现代码 #

from time import sleep
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

URL_alnair = 'https://alnair.ae/app/view/1412/3386/apartment/apartments'
o = Options()
o.add_experimental_option('detach', True)
o.add_argument('--start-maximized')
chrome_driver_path = r"YOUR PATH"
driver = webdriver.Chrome(options=o, service=ChromeService(executable_path=chrome_driver_path))

def get_data():
    driver.get(URL_alnair)
    # 等待滚动容器加载完成（超时10秒）
    wait = WebDriverWait(driver, 10)
    scroll_container = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, '._tableContainer_18734_1')))
    
    # 循环滚动直到没有新内容加载
    last_scroll_height = driver.execute_script("return arguments[0].scrollHeight", scroll_container)
    while True:
        # 将滚动容器滚动到底部
        driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", scroll_container)
        sleep(2)  # 等待新内容加载（可根据页面加载速度调整）
        current_scroll_height = driver.execute_script("return arguments[0].scrollHeight", scroll_container)
        
        # 判断是否还有新内容加载
        if current_scroll_height == last_scroll_height:
            break
        last_scroll_height = current_scroll_height
    
    # 此时可获取所有加载完成的表格行
    all_rows = driver.find_elements(By.CSS_SELECTOR, '._tr_18734_1')
    print(f"成功加载{len(all_rows)}条房源数据")

get_data()

关键说明 #

1. 定位滚动容器：通过CSS_SELECTOR找到表格的父级滚动容器（._tableContainer_18734_1），该容器是实际承载滚动条的元素。
2. 容器内滚动：使用JavaScript直接操作容器的scrollTop属性，将其设置为容器的scrollHeight（即容器内容总高度），实现滚动到底部。
3. 循环加载判断：通过对比滚动前后的容器scrollHeight，判断是否还有新内容加载，避免无限循环。

内容的提问来源于stack exchange，提问作者EchoesofDK