使用Selenium爬取Booking.com仅获51条结果，如何获取全部600+结果？

使用Selenium爬取Booking.com时，页面显示有600多条搜索结果，但代码仅能爬取到25条结果。

用户代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
import pandas as pd

url = 'https://www.booking.com/searchresults.html?aid=397645&label=bin859jc-1DCAEoggI46AdIM1gDaKcBiAEBmAExuAEXyAEM2AED6AEB-AECiAIBqAIDuAK-xqeyBsACAdICJGEwYTAzM2QwLTQ1MjEtNGIzNi04NTYyLTcxMjY5NzJlMGNjNtgCBOACAQ&sid=968beab70293559758c39b5f6e59b26e&checkin=2024-05-20&checkout=2024-05-23&region=5140&'
driver = webdriver.Edge()
driver.get(url)

name = []
location = []

container = driver.find_element(By.CLASS_NAME, "d4924c9e74")
apartment = container.find_elements(By.XPATH, './/div[contains(@role, "group")]')

for x in apartment:
    name.append(x.find_element(By.XPATH, './/a[contains(@data-testid, "title")]').text)
    location.append(x.find_element(By.XPATH, './/span[contains(@data-testid, "address")]').text)
    # print(x.text)

hotels = {
    'name' : name,
    'location' : location
}
print(hotels)

爬取结果 #

{'name': ['The Mora Zanzibar\nOpens in new window', 'Zanziblue\nOpens in new window', 'Maisha Nungwi\nOpens in new window', 'Baladin Zanzibar Beach Hotel\nOpens in new window', 'Be Zanzibar Boutique Hotel\nOpens in new window', 'Promised Land Lodge\nOpens in new window', 'Kwanza Resort by SUNRISE\nOpens in new window', 'Karafuu Beach Resort & Spa\nOpens in new window', 'Hekaya Zanzibar\nOpens in new window', 'Paradise Beach Resort & Spa\nOpens in new window', 'Zuri Zanzibar\nOpens in new window', 'Tembo House Hotel\nOpens in new window', 'Sunset Kendwa Beach Hotel\nOpens in new window', 'Villa Nyota Zanzibar\nOpens in new window', 'Kizikula\nOpens in new window', 'Hotel Riu Jambo - All Inclusive\nOpens in new window', 'Chumbe Island Coral Park\nOpens in new window', 'Gold Zanzibar Beach House & Spa\nOpens in new window', 'Mtende Beach Bungalow océan view\nOpens in new window', 'Le Mersenne Zanzibar, Autograph Collection\nOpens in new window', 'Pongwe Bay Resort\nOpens in new window', 'Aurelia Zanzibar\nOpens in new window', 'Aya Beach Resort\nOpens in new window', 'Z-Lodge Zanzibar\nOpens in new window', 'Mayai Ocean Resort\nOpens in new window'], 'location': ['Matemwe', 'Matemwe', 'Nungwi Beach, Nungwi', 'Michamvi', 'Mfumbwi', 'Kizimkazi Beach, Kizimkazi', 'Kizimkazi', 'Pingwe', 'Pingwe', 'Uroa', 'Kendwa Beach, Kendwa', 'Zanzibar City', 'Kendwa Beach, Kendwa', 'Kidenga', 'Kizimkazi', 'Nungwi', 'Mbweni', 'Kendwa Beach, Kendwa', 'Mtende', 'Michamvi', 'Pongwe', 'Ngambo', 'Kizimkazi Beach, Kizimkazi', 'Kiwengwa', 'Bwejuu Beach, Bwejuu']}

核心原因 #

• 懒加载机制：Booking.com采用滚动加载逻辑，仅初始加载页面可见区域的酒店卡片，未滚动到的区域不会加载内容，导致代码只能获取到初始加载的25条结果。
• 元素定位局限性：通过单一容器d4924c9e74定位子元素，该容器仅包含初始加载的内容，无法覆盖后续滚动加载的元素。

具体修复方案 #

1. 实现滚动加载触发所有内容加载 #

模拟浏览器滚动到底部，重复操作直到页面高度不再变化（即所有内容加载完成）：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import pandas as pd

url = 'https://www.booking.com/searchresults.html?aid=397645&label=bin859jc-1DCAEoggI46AdIM1gDaKcBiAEBmAExuAEXyAEM2AED6AEB-AECiAIBqAIDuAK-xqeyBsACAdICJGEwYTAzM2QwLTQ1MjEtNGIzNi04NTYyLTcxMjY5NzJlMGNjNtgCBOACAQ&sid=968beab70293559758c39b5f6e59b26e&checkin=2024-05-20&checkout=2024-05-23&region=5140&'
driver = webdriver.Edge()
driver.get(url)
driver.maximize_window()  # 避免元素因窗口过小被遮挡

name = []
location = []

# 循环滚动加载所有内容
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(3)  # 等待页面加载新内容
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break  # 高度不变，说明所有内容已加载
    last_height = new_height

# 显式等待所有酒店卡片加载完成
WebDriverWait(driver, 20).until(
    EC.presence_of_all_elements_located((By.XPATH, '//div[contains(@role, "group")]'))
)

# 定位所有酒店元素
apartments = driver.find_elements(By.XPATH, '//div[contains(@role, "group")]')

for x in apartments:
    try:
        # 提取信息，捕获异常防止单个元素问题中断爬取
        hotel_name = x.find_element(By.XPATH, './/a[contains(@data-testid, "title")]').text
        hotel_location = x.find_element(By.XPATH, './/span[contains(@data-testid, "address")]').text
        name.append(hotel_name)
        location.append(hotel_location)
    except Exception as e:
        print(f"提取元素失败: {e}")
        continue

hotels = {
    'name': name,
    'location': location
}
print(f"共爬取 {len(name)} 条结果")
print(hotels)

driver.quit()

2. 优化细节 #

• 显式等待替代固定休眠：使用WebDriverWait等待元素加载，比固定sleep更高效可靠。
• 异常捕获：部分酒店卡片可能存在结构差异，添加异常处理避免爬取中断。
• 直接定位所有目标元素：跳过中间容器，直接定位所有role="group"的元素，避免容器选择错误导致遗漏。

3. 反爬注意事项 #

• 适当延长滚动后的等待时间，或使用随机间隔（如time.sleep(random.uniform(2,4))），避免触发网站反爬机制。
• 可考虑添加User-Agent伪装，模拟真实浏览器请求。

内容的提问来源于stack exchange，提问作者Martha Imoh