Selenium爬亚马逊搜索结果遇NoSuchElementException报错求助

核心原因分析 #

出现NoSuchElementException的常见触发点包括：元素定位表达式失效、页面未完全加载、反爬机制干扰、弹窗/iframe遮挡元素。结合亚马逊英国站的页面特性，以下是针对性解决方法：

分步解决措施 #

1. 修正元素定位表达式 #

亚马逊页面元素会动态迭代，避免使用易失效的绝对XPATH，改用基于稳定属性的相对定位：

• 标题定位：//div[@data-component-type='s-search-result']//h2/a/span（依托亚马逊搜索结果的通用标识data-component-type）
• 价格定位：//div[@data-component-type='s-search-result']//span[@class='a-price-whole']

2. 替换固定等待为显式等待 #

摒弃time.sleep()的固定延迟，改用Selenium显式等待确保元素加载完成：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# 等待标题元素出现，最长等待10秒
title_element = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.XPATH, "//div[@data-component-type='s-search-result']//h2/a/span"))
)

3. 处理页面弹窗与iframe #

• 优先处理亚马逊的Cookie授权弹窗：

  try:
      accept_cookie_btn = WebDriverWait(driver, 5).until(
          EC.element_to_be_clickable((By.ID, "sp-cc-accept"))
      )
      accept_cookie_btn.click()
  except:
      pass  # 无弹窗则跳过
  

• 检查目标元素是否处于iframe内，若存在需先切换iframe：

  driver.switch_to.frame("iframe-id")  # 替换为实际iframe的ID或定位符
  

4. 规避反爬机制 #

亚马逊会检测自动化行为，可通过以下方式模拟真实用户：

• 添加自定义User-Agent：

  from selenium.webdriver.chrome.options import Options
  
  options = Options()
  options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
  driver = webdriver.Chrome(options=options)
  

• 加入随机延迟避免操作过于频繁：

  import random
  time.sleep(random.uniform(2, 5))
  

完整修正代码示例 #

import csv
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options
import time
import random

# 读取关键词CSV
keywords = []
with open('keywords.csv', 'r', encoding='utf-8') as f:
    reader = csv.reader(f)
    for row in reader:
        keywords.extend(row)

# 配置浏览器
options = Options()
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")
driver = webdriver.Chrome(options=options)
driver.get("https://www.amazon.co.uk/")

# 处理Cookie弹窗
try:
    accept_btn = WebDriverWait(driver, 5).until(
        EC.element_to_be_clickable((By.ID, "sp-cc-accept"))
    )
    accept_btn.click()
except:
    pass

# 遍历关键词获取结果
for keyword in keywords:
    # 定位搜索框并执行搜索
    search_box = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "twotabsearchtextbox"))
    )
    search_box.clear()
    search_box.send_keys(keyword)
    search_btn = driver.find_element(By.ID, "nav-search-submit-button")
    search_btn.click()
    
    # 随机延迟等待结果加载
    time.sleep(random.uniform(2, 5))
    
    # 获取前5条搜索结果
    results = WebDriverWait(driver, 10).until(
        EC.presence_of_all_elements_located((By.XPATH, "//div[@data-component-type='s-search-result']"))
    )[:5]
    
    for idx, result in enumerate(results, 1):
        try:
            title = result.find_element(By.XPATH, ".//h2/a/span").text
            price = result.find_element(By.XPATH, ".//span[@class='a-price-whole']").text
            print(f"第{idx}条 - 标题: {title}, 价格: £{price}")
        except NoSuchElementException:
            print(f"第{idx}条 - 部分信息缺失")

driver.quit()

内容的提问来源于stack exchange，提问作者David Wyness