Python中Selenium find_element失效但Beautiful Soup find可用问题排查

问题概述 #

我想用Python爬取亚马逊这款商品页面的价格（目标价格$25.99），用Beautiful Soup写的代码能正常返回结果，但Selenium代码运行后控制台没有任何输出。我原以为Selenium里的find_element(By.CSS_SELECTOR, 'span .a-offscreen')和Beautiful Soup的find(name='span', class_='a-offscreen')作用一致，试过XPATH也没用，想知道问题出在哪。

可用的Beautiful Soup代码 #

这段代码能正常返回价格：

# with beautiful soup
import requests
from bs4 import BeautifulSoup

PRODUCT="https://www.amazon.com/Guffercty-kred-Sublimation-Mechanical-Keyboard/dp/B09HWZQQZJ/ref=sr_1_14?crid=3UHD6OMRY6RYG&keywords=keycaps&qid=1667444474&qu=eyJxc2MiOiI4Ljc5IiwicXNhIjoiOC41OCIsInFzcCI6IjcuOTMifQ%3D%3D&sprefix=keycap%2Caps%2C275&sr=8-14&th=1"

response = requests.get(PRODUCT,
                        headers={"Accept-Language":"ko,en-US;q=0.9,en;q=0.8,sv;q=0.7,ja;q=0.6",
                                 "User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36"})
soup = BeautifulSoup(response.text, "html.parser")

price = float(soup.find(name="span", class_="a-offscreen").getText())
print(price)

控制台输出：25.99

存在问题的Selenium代码 #

这段代码运行后无输出：

# with selenium
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By

service = Service("/my/chrome/driver/path/chromedriver")

driver = webdriver.Chrome(service=service)
driver.get(url="https://www.amazon.com/Guffercty-kred-Sublimation-Mechanical-Keyboard/dp/B09HWZQQZJ/ref=sr_1_14?crid=3UHD6OMRY6RYG&keywords=keycaps&qid=1667444474&qu=eyJxc2MiOiI4Ljc5IiwicXNhIjoiOC41OCIsInFzcCI6IjcuOTMifQ%3D%3D&sprefix=keycap%2Caps%2C275&sr=8-14&th=1")
price = driver.find_element(By.CSS_SELECTOR, 'span .a-offscreen')
print(price.text)

问题原因与解决方法 #

1. CSS选择器错误 #

你写的span .a-offscreen是后代选择器，意思是找所有<span>标签下的、class为a-offscreen的子元素。但目标元素本身就是class为a-offscreen的<span>，所以正确的CSS选择器应该是span.a-offscreen（去掉空格，直接用元素+类选择器），这才和Beautiful Soup的find(name='span', class_='a-offscreen')对应。

2. 隐藏元素无法用.text获取 #

亚马逊的a-offscreen类是用来隐藏文本的（供屏幕阅读器读取），Selenium的.text方法只会返回可见元素的文本内容。对于隐藏元素，需要用get_attribute('textContent')或者get_attribute('innerText')来获取文本。

3. 缺少页面加载等待 #

页面加载需要时间，直接执行查找元素的操作可能会遇到元素还未渲染完成的情况，建议添加显式等待，确保元素存在后再进行操作。

修正后的Selenium代码 #

# 修正后的Selenium代码
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

service = Service("/my/chrome/driver/path/chromedriver")
driver = webdriver.Chrome(service=service)
driver.get("https://www.amazon.com/Guffercty-kred-Sublimation-Mechanical-Keyboard/dp/B09HWZQQZJ/ref=sr_1_14?crid=3UHD6OMRY6RYG&keywords=keycaps&qid=1667444474&qu=eyJxc2MiOiI4Ljc5IiwicXNhIjoiOC41OCIsInFzcCI6IjcuOTMifQ%3D%3D&sprefix=keycap%2Caps%2C275&sr=8-14&th=1")

# 显式等待元素加载，最长等待10秒
price_element = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.CSS_SELECTOR, 'span.a-offscreen'))
)
# 获取隐藏元素的文本内容
price_text = price_element.get_attribute('textContent')
# 去除$符号并转换为浮点数
price = float(price_text.replace('$', ''))
print(price)

内容的提问来源于stack exchange，提问作者Jay