循环爬取Airbnb房源评分时遇索引错误或空列表问题求助

问题背景 #

爬取Airbnb房源的清洁度、位置等评分时，单页面爬取可正常获取数据，但放入多页面循环后，要么出现列表索引越界错误，要么返回空列表。

现有代码详情 #

导入模块 #

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
from bs4 import BeautifulSoup
import pandas as pd
import time
import re
import requests
from datetime import date

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.maximize_window()

单页可行爬取方法 #

方法一（直接索引定位）

单页运行正常，放入循环后出现索引越界：

wait = WebDriverWait(driver,20)
test_url = 'https://www.airbnb.com/rooms/51833563?adults=1&children=0&infants=0&check_in=2022-09-18&check_out=2022-09-23&federated_search_id=96d13170-4935-492e-ad71-ffe0c39bc006&source_impression_id=p3_1663079036_%2FKJDqmaFlIk3h8a6'
driver.get(test_url)
time.sleep(3)
soup2 = BeautifulSoup(driver.page_source, 'lxml')
scores = []
Cleanliness=  driver.find_elements(By.XPATH,"//span[@class='_4oybiu'][1]")[0].text
Accuracy= driver.find_elements(By.XPATH,"//span[@class='_4oybiu'][1]")[1].text
Communication=  driver.find_elements(By.XPATH,"//span[@class='_4oybiu'][1]")[2].text
Location= driver.find_elements(By.XPATH,"//span[@class='_4oybiu'][1]")[3].text
Check_in= driver.find_elements(By.XPATH,"//span[@class='_4oybiu'][1]")[4].text
Value= driver.find_elements(By.XPATH,"//span[@class='_4oybiu'][1]")[5].text

方法二（类名批量获取）

单页运行正常，放入循环后返回空列表：

wait = WebDriverWait(driver,20)
test_url = 'https://www.airbnb.com/rooms/51833563?adults=1&children=0&infants=0&check_in=2022-09-18&check_out=2022-09-23&federated_search_id=96d13170-4935-492e-ad71-ffe0c39bc006&source_impression_id=p3_1663079036_%2FKJDqmaFlIk3h8a6'
driver.get(test_url)
time.sleep(3)
soup2 = BeautifulSoup(driver.page_source, 'lxml')
scores = []
Cleanliness=  driver.find_elements(By.CLASS_NAME,"_4oybiu")
for i in Cleanliness:
    scores.append(i.text)

循环爬取代码（出现问题的版本） #

url = 'https://www.airbnb.com/s/Thessaloniki--Greece/homes?tab_id=home_tab&flexible_trip_lengths%5B%5D=one_week&refinement_paths%5B%5D=%2Fhomes&place_id=ChIJ7eAoFPQ4qBQRqXTVuBXnugk&query=Thessaloniki%2C%20Greece&date_picker_type=calendar&search_type=user_map_move&price_filter_input_type=0&ne_lat=40.66256734970964&ne_lng=23.003752862853986&sw_lat=40.59051931897441&sw_lng=22.892087137145978&zoom=13&search_by_map=true&federated_search_session_id=1ed21e1c-0c5e-4529-ab84-267361eac02b&pagination_search=true&items_offset={offset}&section_offset=2'

p_lat = re.compile(r'"lat":([-0-9.]+),')
p_lng = re.compile(r'"lng":([-0-9.]+),')

data = []
calendar = []

for offset in range(0, 20, 20):
    print('offset:', offset)
    
    driver.get(url.format(offset=offset))
    time.sleep(2)
    
    soup = BeautifulSoup(driver.page_source, 'lxml')

    detailed_pages = []
    
    for card in soup.select('div[class="c4mnd7m dir dir-ltr"] a[class="ln2bl2p dir dir-ltr"]'):
        link = 'https://www.airbnb.com' + card['href']
        detailed_pages.append(link)
        
    print('len(detailed_pages):', len(detailed_pages))
    
    for number, page in enumerate(detailed_pages, 1):
        print(number, 'page:', page)

        driver.get(page)
        time.sleep(5)
        
        soup2 = BeautifulSoup(driver.page_source, 'lxml')
        Cleanliness=  driver.find_elements(By.XPATH,"//span[@class='_4oybiu'][1]")[0].text
        Accuracy= driver.find_elements(By.XPATH,"//span[@class='_4oybiu'][1]")[1].text
        Communication=  driver.find_elements(By.XPATH,"//span[@class='_4oybiu'][1]")[2].text
        Location= driver.find_elements(By.XPATH,"//span[@class='_4oybiu'][1]")[3].text
        Check_in= driver.find_elements(By.XPATH,"//span[@class='_4oybiu'][1]")[4].text
        Value= driver.find_elements(By.XPATH,"//span[@class='_4oybiu'][1]")[5].text

问题原因及解决方案 #

核心原因 #

1. 固定等待不可靠：time.sleep()的固定时长无法适配不同页面的加载速度，循环中部分页面加载慢时，元素还未渲染完成就执行获取操作，导致找不到元素。
2. 未处理无评分房源：部分房源可能没有用户评分，直接按索引取值会触发越界错误。
3. 元素定位不稳定：Airbnb的class名称（如_4oybiu）可能动态变化，或页面结构调整导致定位失效。

修复方案 #

1. 用显式等待替代固定sleep

使用WebDriverWait等待评分元素加载完成，确保元素存在后再获取。

2. 增加异常处理与空值判断

对无评分的房源返回默认值（如None），避免索引越界。

3. 采用更稳定的定位方式

通过评分项的标签文本（如"Cleanliness"）关联对应的分数，减少对动态class的依赖。

修改后的循环爬取代码 #

url = 'https://www.airbnb.com/s/Thessaloniki--Greece/homes?tab_id=home_tab&flexible_trip_lengths%5B%5D=one_week&refinement_paths%5B%5D=%2Fhomes&place_id=ChIJ7eAoFPQ4qBQRqXTVuBXnugk&query=Thessaloniki%2C%20Greece&date_picker_type=calendar&search_type=user_map_move&price_filter_input_type=0&ne_lat=40.66256734970964&ne_lng=23.003752862853986&sw_lat=40.59051931897441&sw_lng=22.892087137145978&zoom=13&search_by_map=true&federated_search_session_id=1ed21e1c-0c5e-4529-ab84-267361eac02b&pagination_search=true&items_offset={offset}&section_offset=2'

p_lat = re.compile(r'"lat":([-0-9.]+),')
p_lng = re.compile(r'"lng":([-0-9.]+),')

data = []
wait = WebDriverWait(driver, 20)

for offset in range(0, 20, 20):
    print('offset:', offset)
    
    driver.get(url.format(offset=offset))
    # 等待列表页加载完成
    wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'div[class="c4mnd7m dir dir-ltr"]')))
    
    soup = BeautifulSoup(driver.page_source, 'lxml')
    detailed_pages = []
    
    for card in soup.select('div[class="c4mnd7m dir dir-ltr"] a[class="ln2bl2p dir dir-ltr"]'):
        link = 'https://www.airbnb.com' + card['href']
        detailed_pages.append(link)
        
    print('len(detailed_pages):', len(detailed_pages))
    
    for number, page in enumerate(detailed_pages, 1):
        print(number, 'page:', page)
        driver.get(page)
        
        # 初始化评分默认值
        score_dict = {
            'Cleanliness': None,
            'Accuracy': None,
            'Communication': None,
            'Location': None,
            'Check_in': None,
            'Value': None
        }
        
        try:
            # 等待评分区域加载
            wait.until(EC.presence_of_element_located((By.XPATH, '//div[contains(@class, "_1hxyyw3")]')))
            
            # 通过标签文本定位对应分数
            for key in score_dict.keys():
                # 定位评分标签，再取相邻的分数元素
                score_element = wait.until(EC.presence_of_element_located(
                    (By.XPATH, f'//span[text()="{key}"]/following-sibling::span')
                ))
                score_dict[key] = score_element.text
                
        except Exception as e:
            print(f"页面 {page} 评分获取失败: {str(e)}")
            
        data.append(score_dict)

# 转换为DataFrame查看
df = pd.DataFrame(data)
print(df)
driver.quit()

内容的提问来源于stack exchange，提问作者Lefteris Kyprianou