如何修复Python爬虫（Selenium+BeautifulSoup）的数组长度不一致错误

问题根源 #

1. 分离遍历导致数据错位：原代码先遍历所有价格元素，再遍历所有酒店元素，两个选择器匹配的元素数量可能不一致（比如部分酒店无显示价格、选择器误匹配无关元素），直接导致hotels/links和prices列表长度不同。
2. 页面offset未更新：原代码中page变量固定为初始url，offset递增后未重新生成带新offset的链接，始终爬取同一页面，数据重复且逻辑错误。
3. 长度修正逻辑无效：最后取最小长度但创建DataFrame时仍使用原列表，未对过长列表做截断，依然触发长度不匹配报错。

修复方案 #

核心思路：以单个酒店卡片为单位遍历，在每个卡片容器内同时提取名称、链接、价格，确保每条数据一一对应；同时正确更新页面offset，修复循环爬取逻辑。

修复后代码 #

import bs4
import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver
import time

# 用Selenium获取页面源码
def getPageBySel(url):
    options = webdriver.ChromeOptions()
    options.add_argument("--disable-dev-shm-usage")
    options.add_argument("--remote-debugging-port=9222")
    options.add_argument("--window-size=1920x1080")
    driver = webdriver.Chrome(options=options)
    driver.get(url)
    page = driver.page_source
    driver.quit()
    return page

# 改为列表（集合无序，遍历顺序不可控）
hotelsArr = [
    f'https://www.booking.com/searchresults.he.html?aid=397594&label=gog235jc-1DCAEoggI46AdIDlgDaGqIAQGYAQ64ARfIAQzYAQPoAQH4AQKIAgGoAgO4ArvKrqIGwAIB0gIkMWJjMjhhNzItNDZhNC00NDZmLTk1YzgtNjhiOWM0NmM0NDA42AIE4AIB&dest_id=-2601889&dest_type=city&group_adults=2&req_adults=2&no_rooms=1&group_children=0&checkin=2024-01-04&checkout=2024-01-07&req_children=0&offset={0}',
    f'https://www.booking.com/searchresults.he.html?ss=Manchester%2C+Greater+Manchester%2C+United+Kingdom&ssne=%D7%9E%D7%A0%D7%A6%27%D7%A1%D7%98%D7%A8&ssne_untouched=%D7%9E%D7%A0%D7%A6%27%D7%A1%D7%98%D7%A8&efdco=1&label=gog235jc-1DCAEoggI46AdIDlgDaGqIAQGYAQ64ARfIAQzYAQPoAQH4AQKIAgGoAgO4Ar3Xs6IGwAIB0gIkMWU3MTc2OTUtZDZkNi00NzFhLTk2NWYtMDczNjk5MDNhN2U52AIE4AIB&aid=397594&lang=he&sb=1&src_elem=sb&src=index&dest_id=-2602512&dest_type=city&ac_position=0&ac_click_type=b&ac_langcode=en&ac_suggestion_list_length=5&search_selected=true&search_pageview_id=39e046de839803a2&ac_meta=GhAzOWUwNDZkZTgzOTgwM2EyIAAoATICZW46Ck1hbmNoZXN0ZXJAAEoAUAA%3D&group_adults=2&checkin=2024-01-04&checkout=2024-01-07&no_rooms=1&group_children=0&sb_travel_purpose=leisure&offset={0}',
    f'https://www.booking.com/searchresults.he.html?aid=7961375&lang=he&sid=ff4607c90e3e0d79763672e65389c94b&sb=1&sb_lp=1&src=index&src_elem=sb&error_url=https%3A%2F%2Fwww.booking.com%2Findex.he.html%3Faid%3D7961375%26sid%3Dff4607c90e3e0d79763672e65389c94b%26sb_price_type%3Dtotal%26%26&ss=Liverpool%2C+Merseyside%2C+United+Kingdom&is_ski_area=&checkin_year=&checkin_month=&checkout_year=&checkout_month=&efdco=1&group_adults=2&group_children=0&no_rooms=1&b_h4u_keep_filters=&from_sf=1&ss_raw=Liverpool&ac_position=0&ac_langcode=en&ac_click_type=b&ac_meta=GhBhNTQ5NGFhODM2MTgwMjFkIAAoATICZW46CUxpdmVycG9vbEAASgBQAA%3D%3D&dest_id=-2601422&dest_type=city&iata=LPL&place_id_lat=53.4109&place_id_lon=-2.97811&search_pageview_id=a5494aa83618021d&search_selected=true&search_pageview_id=a5494aa83618021d&checkin=2024-01-04&checkout=2024-01-07&ac_suggestion_list_length=5&ac_suggestion_theme_list_length=0&offset={0}',
    f'https://www.booking.com/searchresults.he.html?ss=Birmingham%2C+West+Midlands%2C+United+Kingdom&ssne=%D7%91%D7%A8%D7%9E%D7%99%D7%A0%D7%92%D7%94%D7%90%D7%9D&ssne_untouched=%D7%91%D7%A8%D7%9E%D7%99%D7%A0%D7%92%D7%94%D7%90%D7%9D&efdco=1&label=gen173nr-1BCAEoggI46AdIM1gEaGqIAQGYAQ64ARfIAQzYAQHoAQGIAgGoAgO4AsL2s6IGwAIB0gIkM2VlZjk2YjEtMDJhYi00YmExLTg1NmEtOTIxYTNhNzdhMWQ22AIF4AIB&sid=ff4607c90e3e0d79763672e65389c94b&aid=304142&lang=he&sb=1&src_elem=sb&src=index&dest_id=-2589989&dest_type=city&ac_position=0&ac_click_type=b&ac_langcode=en&ac_suggestion_list_length=5&search_selected=true&search_pageview_id=acea4ea102160498&ac_meta=GhBhY2VhNGVhMTAyMTYwNDk4IAAoATICZW46BWJpcm1pQABKAFAA&group_adults=2&checkin=2024-01-04&checkout=2024-01-07&no_rooms=1&group_children=0&sb_travel_purpose=leisure&offset={0}',
    f'https://www.booking.com/searchresults.he.html?ss=%D7%90%D7%93%D7%99%D7%A0%D7%91%D7%95%D7%A8%D7%95%2C+%D7%A1%D7%A7%D7%95%D7%98%D7%9C%D7%A0%D7%93%2C+%D7%91%D7%A8%D7%99%D7%98%D7%A0%D7%99%D7%94&ssne=%D7%92%D7%9C%D7%90%D7%96%D7%92%D7%95&ssne_untouched=%D7%92%D7%9C%D7%90%D7%96%D7%92%D7%95&efdco=1&label=gen173nr-1BCAEoggI46AdIM1gEaGqIAQGYAQ64ARfIAQzYAQHoAQGIAgGoAgO4AoP4s6IGwAIB0gIkZmQ0YjhjNTMtZTc2ZS00NDZkLThmMmEtNmUyZDk3YTAwZWJl2AIF4AIB&sid=ff4607c90e3e0d79763672e65389c94b&aid=304142&lang=he&sb=1&src_elem=sb&src=index&dest_id=-2595386&dest_type=city&ac_position=0&ac_click_type=b&ac_langcode=he&ac_suggestion_list_length=5&search_selected=true&search_pageview_id=ec504f01a010003c&ac_meta=GhBlYzUwNGYwMWEwMTAwMDNjIAAoATICaGU6A2VkaUAASgBQAA%3D%3D&group_adults=2&checkin=2024-01-04&checkout=2024-01-07&no_rooms=1&group_children=0&sb_travel_purpose=leisure&offset={0}',
    f'https://www.booking.com/searchresults.he.html?ss=%D7%99%D7%95%D7%A8%D7%A7&ssne=%D7%99%D7%95%D7%A8%D7%A7&ssne_untouched=%D7%99%D7%95%D7%A8%D7%A7&label=gen173nr-1BCAEoggI46AdIM1gEaGqIAQGYAQ64ARfIAQzYAQHoAQGIAgGoAgO4Auj6s6IGwAIB0gIkM2IwYTU3ODgtNDJiYS00ZDk0LWI0MDAtNGU3M2U3ZDlkNzM42AIF4AIB&sid=ff4607c90e3e0d79763672e65389c94b&aid=304142&lang=he&sb=1&src_elem=sb&src=index&dest_id=-2612321&dest_type=city&group_adults=2&no_rooms=1&group_children=0&sb_travel_purpose=leisure&offset={0}'
]

hotels = []
links = []
prices = []

# 遍历每个城市的搜索链接
for base_url in hotelsArr:
    offset = 0
    # 爬取前40页（offset从0到975，步长25）
    while offset <= 975:
        # 生成带当前offset的页面url
        current_url = base_url.replace(f'offset={0}', f'offset={offset}')
        
        # 等待页面加载完成
        while True:
            temp = getPageBySel(current_url)
            soup = BeautifulSoup(temp, 'html.parser')
            # 检查酒店卡片是否加载
            hotel_cards = soup.select('div[data-testid="property-card"]')
            if len(hotel_cards) > 0:
                break
            time.sleep(1.5)
        
        # 遍历每个酒店卡片，同时提取所有信息
        for card in hotel_cards:
            # 提取酒店名称
            name_elem = card.select_one('h3.a4225678b2 .fcab3ed991.a23c043802')
            name = name_elem.get_text(strip=True) if name_elem else 'N/A'
            
            # 提取酒店链接
            link_elem = card.select_one('h3.a4225678b2 a')
            link = link_elem['href'] if link_elem else 'N/A'
            
            # 提取价格
            price_elem = card.select_one('.fcab3ed991.fbd1d3018c.e729ed5ab6')
            price = price_elem.get_text(strip=True) if price_elem else 'N/A'
            
            # 添加到列表
            hotels.append(name)
            links.append(link)
            prices.append(price)
        
        offset += 25

# 创建DataFrame（此时三个列表长度必然一致）
df = pd.DataFrame({
    'Hotel': hotels,
    'Link': links,
    'Prices': prices
})

print(df)
df.to_csv('hotels_list30.csv', index=True)

关键修复点 #

• 统一遍历单位：以酒店卡片容器div[data-testid="property-card"]为单位，确保每个卡片对应一条数据，三个列表同步增长。
• 修复offset更新：每次循环生成带新offset的url，确保爬取不同页面。
• 替换集合为列表：保证城市遍历顺序稳定，避免数据混乱。
• 增强元素检查：对每个字段的元素存在性做判断，避免空指针错误，同时填充默认值。

内容的提问来源于stack exchange，提问作者Gilad Levy