Selenium爬取Cargurus页面导航重复数据问题求助

核心原因分析 #

Cargurus是动态渲染网站，换页后内容依赖AJAX异步加载，若未等待新内容完全加载就抓取，会获取旧DOM缓存；直接修改URL可能触发网站会话缓存，导致页面未真正刷新。

具体解决步骤 #

1. 等待新内容加载完成 #

点击下页按钮或跳转URL后，必须等待关键元素（如车辆列表最后一条内容、当前页码）更新，避免提前抓取旧内容。示例代码片段：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

# 点击下页前记录当前页最后一辆车的内容
current_last_car = driver.find_elements(By.CSS_SELECTOR, 'div[data-test="vehicle-listing"]')[-1].text
next_btn = driver.find_element(By.CSS_SELECTOR, 'a[data-test="next-page"]')
next_btn.click()

# 等待最后一辆车内容变化，超时10秒
WebDriverWait(driver, 10).until(
    lambda d: d.find_elements(By.CSS_SELECTOR, 'div[data-test="vehicle-listing"]')[-1].text != current_last_car
)

2. 优先使用页面内分页按钮 #

直接修改URL易被缓存拦截，优先点击页面原生的「下一页」按钮，确保前端状态同步更新。

3. 强制刷新页面（URL跳转场景） #

若必须用URL跳转，跳转后执行强制刷新绕过缓存：

driver.get(new_page_url)
# 强制刷新，忽略本地缓存
driver.execute_script("location.reload(true);")
# 等待车辆列表重新加载
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, 'div[data-test="vehicle-listing"]'))
)

4. 校验元素定位准确性 #

确保车辆条目的选择器定位到当前页面的动态列表，避免使用固定缓存容器的定位符。可通过浏览器开发者工具确认选择器（比如div[data-test="vehicle-listing"]）。

5. 规避反爬限制 #

Cargurus会检测爬虫行为，可添加以下措施：

• 每次操作后添加1-3秒随机延迟：import random; time.sleep(random.uniform(1,3))
• 设置真实浏览器UA，替换默认Selenium标识：

from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")
driver = webdriver.Chrome(options=options)

完整修正代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import random

def scrape_cargurus_ferrari():
    options = webdriver.ChromeOptions()
    options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")
    driver = webdriver.Chrome(options=options)
    driver.get("https://www.cargurus.com/Cars/inventorylisting/viewDetailsFilterViewInventoryListing.action?sourceContext=carGurusHomePageModel&entitySelectingHelper.selectedEntity=Ferrari")
    
    all_cars = []
    max_pages = 3  # 示例爬取3页
    
    for page in range(max_pages):
        # 等待当前页车辆列表加载完成
        WebDriverWait(driver, 10).until(
            EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'div[data-test="vehicle-listing"]'))
        )
        
        # 抓取当前页车辆信息
        cars = driver.find_elements(By.CSS_SELECTOR, 'div[data-test="vehicle-listing"]')
        for car in cars:
            car_info = {
                "title": car.find_element(By.CSS_SELECTOR, 'h4[data-test="listing-title"]').text,
                "price": car.find_element(By.CSS_SELECTOR, 'span[data-test="listing-price"]').text,
                "mileage": car.find_element(By.CSS_SELECTOR, 'div[data-test="listing-mileage"]').text
            }
            all_cars.append(car_info)
            print(car_info)
        
        # 切换到下一页（非最后一页时）
        if page < max_pages - 1:
            try:
                current_last_car = cars[-1].text
                next_btn = WebDriverWait(driver, 10).until(
                    EC.element_to_be_clickable((By.CSS_SELECTOR, 'a[data-test="next-page"]'))
                )
                next_btn.click()
                # 等待新页面内容更新
                WebDriverWait(driver, 10).until(
                    lambda d: d.find_elements(By.CSS_SELECTOR, 'div[data-test="vehicle-listing"]')[-1].text != current_last_car
                )
                # 模拟人类操作延迟
                time.sleep(random.uniform(1.5, 3))
            except Exception as e:
                print(f"下一页切换失败: {e}")
                break
    
    driver.quit()
    return all_cars

if __name__ == "__main__":
    scrape_cargurus_ferrari()

内容的提问来源于stack exchange，提问作者ssennna_812