Selenium滚动加载页面爬取商品时偶发漏取元素的问题求解

问题描述 #

尝试爬取Tamimi Markets官网生鲜分类（https://shop.tamimimarkets.com/category/fresh）下所有商品，网站采用滚动加载模式，初始仅加载20个商品，需向下滚动才能加载更多。编写的Selenium代码连续运行5次仅2次成功获取全部商品（2721个），其余次数获取的商品数量差异极大，输出结果为：1136、2721、2721、498、1398。

原代码 #

products_details = []
market_page = "https://shop.tamimimarkets.com"
website = "https://shop.tamimimarkets.com/category/fresh"
def Find_all_products(URL):
    options = Options()
    options.headless = True
    driver = webdriver.Chrome('/chromeDriver', options=options)
    driver.get(URL)
    time.sleep(3)
    last_height = driver.execute_script("return document.body.scrollHeight")
    while True:
    # Scroll down to bottom
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight - 1300);")
    # Wait to load page
        time.sleep(3)
    # Calculate new scroll height and compare with last scroll height
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            break
        last_height = new_height
    
    page_source = driver.page_source
    soup = BeautifulSoup(page_source, 'lxml')
    product = soup.find_all("a",{'class':'yDbmR'})
    for p in product:
        product_URL = market_page + p.attrs['href']
        products_details.append(product_URL)

for _ in range(5):
   Find_all_products(website)
   print(len(products_details))
   products_details.clear()

输出结果 #

1136
2721
2721
498
1398

解决方案 #

1. 替换固定等待为显式等待 #

固定time.sleep()无法适配网络波动，改用Selenium显式等待，精准等待商品元素或滚动后的新内容加载完成：

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# 初始页面加载等待
WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.CSS_SELECTOR, "a.yDbmR"))
)

# 滚动后的等待逻辑替换原time.sleep(3)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
# 等待滚动高度变化或超时
WebDriverWait(driver, 8).until(
    lambda d: d.execute_script("return document.body.scrollHeight") > last_height
)

2. 优化滚动终止条件 #

原逻辑一次滚动高度不变就停止，容易因加载延迟误判，改为连续多次滚动高度不变才终止：

last_height = driver.execute_script("return document.body.scrollHeight")
scroll_stable_count = 0
max_stable_attempts = 3  # 连续3次高度不变则停止

while scroll_stable_count < max_stable_attempts:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    # 等待加载
    WebDriverWait(driver, 8).until(
        lambda d: d.execute_script("return document.body.scrollHeight") != last_height or scroll_stable_count >= max_stable_attempts
    )
    new_height = driver.execute_script("return document.body.scrollHeight")
    
    if new_height == last_height:
        scroll_stable_count += 1
    else:
        scroll_stable_count = 0
        last_height = new_height

3. 修复Headless模式渲染问题 #

Headless模式默认视口较小，可能导致商品无法触发加载，设置桌面级窗口大小：

options = Options()
options.headless = True
options.add_argument("--window-size=1920,1080")  # 模拟桌面浏览器视口
driver = webdriver.Chrome('/chromeDriver', options=options)

4. 增加商品数量验证 #

滚动结束后检查商品数量是否稳定，若未达到预期可二次触发滚动：

def get_current_product_count(driver):
    return len(driver.find_elements(By.CSS_SELECTOR, "a.yDbmR"))

# 滚动结束后验证
initial_count = get_current_product_count(driver)
WebDriverWait(driver, 12).until(lambda d: get_current_product_count(d) == initial_count)
# 若仍有加载可能，再滚动一次
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(2)

5. 启用无痕模式避免缓存干扰 #

缓存可能导致页面加载异常，启用无痕模式并禁用缓存：

options.add_argument("--incognito")
options.add_argument("--disable-cache")
options.add_argument("--disable-application-cache")

内容的提问来源于stack exchange，提问作者Z14231