求助：如何用BeautifulSoup从Kayak抓取最优、低价航班的View Deal链接与价格

代码核心问题 #

1. User-Agent参数错误：设置--user-agent时多了一个末尾双引号，导致UA格式失效，可能触发Kayak反爬或页面渲染异常。
2. 等待方式不可靠：固定时长sleep无法保证动态页面元素完全加载，容易出现元素未就绪就抓取的情况。
3. XPath定位不准确：全局匹配booking-link下的价格，可能抓取到非目标卡片元素，或因Kayak类名动态更新导致定位失败。
4. 反爬配置不足：缺少针对Selenium自动化的反检测设置，易被Kayak拦截。

修正后的完整代码 #

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import random

url = "https://www.kayak.com/flights/AMS-WMI,nearby/2023-02-15/WMI-SOF,nearby/2023-02-18/SOF-BEG,nearby/2023-02-20/BEG-MIL,nearby/2023-02-23/MIL-AMS,nearby/2023-02-25/?sort=bestflight_a"
requests = 0

chrome_options = webdriver.ChromeOptions()
agents = ["Firefox/66.0.3","Chrome/73.0.3683.68","Edge/16.16299"]
selected_agent = agents[requests % len(agents)]
print(f"User agent: {selected_agent}")
# 修复UA的引号问题
chrome_options.add_argument(f'--user-agent={selected_agent}')
# 增强反爬配置
chrome_options.add_experimental_option('useAutomationExtension', False)
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_argument('--disable-blink-features=AutomationControlled')
chrome_options.add_argument('--disable-extensions')
chrome_options.add_argument('--start-maximized')

# 新版本Selenium用Service指定驱动路径
service = Service('/Users/etc./etc./chromedriver')  # 替换为你的chromedriver实际路径
driver = webdriver.Chrome(service=service, options=chrome_options)

try:
    driver.get(url)
    # 处理可能出现的Cookie弹窗
    try:
        cookie_btn = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.XPATH, '//button[contains(text(), "Accept")]'))
        )
        cookie_btn.click()
    except:
        pass  # 无弹窗则跳过

    # 等待前两个目标航班卡片加载完成
    WebDriverWait(driver, 15).until(
        EC.presence_of_all_elements_located((By.XPATH, '//div[contains(@class, "above-button") and (contains(text(), "Best") or contains(text(), "Cheapest"))]/ancestor::div[contains(@class, "nrc6")]'))
    )

    # 定位前两个航班卡片
    target_cards = driver.find_elements(By.XPATH, '//div[contains(@class, "above-button") and (contains(text(), "Best") or contains(text(), "Cheapest"))]/ancestor::div[contains(@class, "nrc6")]')[:2]

    result = []
    for card in target_cards:
        # 获取价格
        price_elem = card.find_element(By.XPATH, './/span[@class="price option-text"]')
        price = int(price_elem.text.replace('$', ''))
        # 获取View Deal链接
        deal_link = card.find_element(By.XPATH, './/a[@class="booking-link"]').get_attribute('href')
        # 获取卡片类型
        card_type = card.find_element(By.XPATH, './/div[contains(@class, "above-button")]').text.strip()
        result.append({
            "type": card_type,
            "price": price,
            "deal_link": deal_link
        })

    print("抓取结果：")
    for item in result:
        print(f"{item['type']}航班 - 价格: ${item['price']}, 链接: {item['deal_link']}")

finally:
    driver.quit()

关键修改说明 #

• 修复UA参数：移除末尾多余双引号，确保UA格式正确。
• 强化反爬：添加禁用自动化检测的配置，降低被Kayak识别为爬虫的概率。
• 显式等待：用WebDriverWait替代固定sleep，确保元素加载完成后再抓取，避免空列表问题。
• 精准定位：先通过"Best"/"Cheapest"标识锁定目标卡片，再在卡片内部抓取价格和链接，避免无关元素干扰。
• 异常处理：增加Cookie弹窗处理和driver销毁逻辑，提升代码稳定性。

内容的提问来源于stack exchange，提问作者June Smith