无限滚动页面的While循环中WebDriver.Wait()失效问题求助

你的代码里用EC.presence_of_all_elements_located((By.CLASS_NAME, 'pb-xl'))作为滚动后的等待条件，这根本起不到等待新内容加载的作用——因为页面一开始就存在这个类的元素，只要元素存在就会触发等待结束，不会等新内容加载完成。

下面给你几个靠谱的修改方案：

方案1：等待滚动高度变化 #

滚动后新内容加载会让页面的scrollHeight变大，我们可以写一个自定义等待条件，直到scrollHeight超过滚动前的数值：

try:
    driver.get(url)
    WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'pb-xl')))
    self.writeToFile('test.txt','Page is loaded\n')
    # 获取初始滚动高度
    last_height = driver.execute_script("return document.body.scrollHeight")

    while True:
        print("Scrolling down...")
        # 滚动到页面底部
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        print("Wait for new content to load...")
        
        # 等待滚动高度变化，超时时间10秒
        try:
            WebDriverWait(driver, 10).until(
                lambda d: d.execute_script("return document.body.scrollHeight") > last_height
            )
        except TimeoutException:
            # 超时说明没有新内容，退出循环
            break
        
        # 计算新的滚动高度并对比
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            break
        print("Loaded new content, updating scroll height")
        last_height = new_height

    self.writeToFile('page_source.txt',driver.page_source)
except TimeoutException:
        self.writeToFile('test.txt','Timed out waiting for initial page load\n')
except Exception as e:
        self.writeToFile('test.txt',f'Error: {str(e)}\n')

方案2：等待元素数量增加 #

如果pb-xl是每个内容项的类名，滚动后新内容会增加这个类的元素数量，我们可以记录滚动前的元素数量，等待数量变多：

try:
    driver.get(url)
    WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'pb-xl')))
    self.writeToFile('test.txt','Page is loaded\n')
    # 获取初始元素数量
    last_element_count = len(driver.find_elements(By.CLASS_NAME, 'pb-xl'))
    last_height = driver.execute_script("return document.body.scrollHeight")

    while True:
        print("Scrolling down...")
        # 滚动到页面底部
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        print("Wait for new content to load...")
        
        # 等待元素数量增加，超时10秒
        try:
            WebDriverWait(driver, 10).until(
                lambda d: len(d.find_elements(By.CLASS_NAME, 'pb-xl')) > last_element_count
            )
        except TimeoutException:
            break
        
        # 更新元素数量和滚动高度
        new_element_count = len(driver.find_elements(By.CLASS_NAME, 'pb-xl'))
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height and new_element_count == last_element_count:
            break
        print(f"Loaded {new_element_count - last_element_count} new items")
        last_element_count = new_element_count
        last_height = new_height

    self.writeToFile('page_source.txt',driver.page_source)
except TimeoutException:
        self.writeToFile('test.txt','Timed out waiting for initial page load\n')
except Exception as e:
        self.writeToFile('test.txt',f'Error: {str(e)}\n')

方案3：等待加载动画消失（如果页面有） #

如果页面滚动后会显示加载动画（比如class为loading-spinner的元素），可以等待这个元素消失：

try:
    driver.get(url)
    WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'pb-xl')))
    self.writeToFile('test.txt','Page is loaded\n')
    last_height = driver.execute_script("return document.body.scrollHeight")

    while True:
        print("Scrolling down...")
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        print("Wait for new content to load...")
        
        # 等待加载动画消失，这里假设加载动画的class是loading-spinner
        try:
            WebDriverWait(driver, 10).until(
                EC.invisibility_of_element_located((By.CLASS_NAME, 'loading-spinner'))
            )
        except TimeoutException:
            break
        
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            break
        print("New content loaded")
        last_height = new_height

    self.writeToFile('page_source.txt',driver.page_source)
except TimeoutException:
        self.writeToFile('test.txt','Timed out waiting for initial page load\n')
except Exception as e:
        self.writeToFile('test.txt',f'Error: {str(e)}\n')

注意事项 #

• 方案1是最通用的，只要页面滚动后内容加载会改变scrollHeight就有效；
• 方案2适合内容项明确用同一个类的场景；
• 方案3需要页面有明确的加载指示器，准确性最高。

内容的提问来源于stack exchange，提问作者Waqas Ahmed