使用Selenium爬取亚马逊商品价格失败，寻求技术帮助

我尝试用Selenium爬取亚马逊商品页面的价格，当前代码如下：

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait as wait

PATH="C:/Users/mrcdi/Downloads/chromedriver-win64/chromedriver-win64/chromedriver.exe"
driver = webdriver.Chrome(PATH)
url="https://www.amazon.com/dp/B00CDD2GU4"
driver.get(url)
price=driver.find_element_by_class_name("a-offscreen").text
print("here is the price ", price)

运行后返回结果如下：

DevTools listening on ws://127.0.0.1:51372/devtools/browser/425a41bc-518b-495b-b70a-72a7e6952f5a 
here is the price 

对应的HTML代码截图：

问题原因及解决方案 #

问题分析

1. 定位元素错误：a-offscreen是亚马逊为屏幕阅读器设置的隐藏元素，其text属性为空，实际显示的价格在可见的a-price容器的子元素中。
2. 未等待页面加载：直接调用find_element时，页面可能未完全渲染完成，导致获取不到有效内容。

修改后的代码

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

PATH = "C:/Users/mrcdi/Downloads/chromedriver-win64/chromedriver-win64/chromedriver.exe"
driver = webdriver.Chrome(PATH)
url = "https://www.amazon.com/dp/B00CDD2GU4"
driver.get(url)

try:
    # 等待价格容器加载完成，最长等待10秒
    price_container = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "a-price"))
    )
    # 获取价格的整数和小数部分并拼接
    price_whole = price_container.find_element(By.CLASS_NAME, "a-price-whole").text
    price_fraction = price_container.find_element(By.CLASS_NAME, "a-price-fraction").text
    full_price = f"{price_whole}.{price_fraction}"
    print("here is the price ", full_price)
finally:
    # 无论成功失败都关闭浏览器
    driver.quit()

额外优化建议

• 自动管理驱动版本：使用webdriver-manager自动匹配ChromeDriver版本，避免手动指定路径的版本兼容问题，替换驱动初始化代码为：

from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

• 反爬应对：亚马逊有反爬机制，频繁访问会触发人机验证，建议添加随机等待时间，或使用代理IP。

内容的提问来源于stack exchange，提问作者Marco Di Giacomo