使用Selenium+BeautifulSoup爬取电商站点时价格字段返回None问题求助

问题核心原因 #

你用Selenium完成邮编提交后，站点会把邮编验证状态存在当前浏览器会话的Cookie里，但你后续抓取页面用的是独立的requests库，requests不会自动携带Selenium浏览器会话里的Cookie，站在站点视角，你用requests发的请求都是未提交过邮编的新访客，自然返回的页面没有价格字段。

解决方案 #

方案1：全程使用Selenium抓取（兼容性最高） #

无需混用requests，所有页面访问、元素提取都用Selenium完成，直接复用已经提交过邮编的浏览器会话，修正后参考代码：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time

driver = webdriver.Chrome(executable_path = "D:\chromedriver\chromedriver.exe")
driver.maximize_window()
wait = WebDriverWait(driver, 30)
baseurl = 'https://www.russellhendrix.com'

# 提交邮编的原有逻辑保持不变
driver.get("https://www.russellhendrix.com/category/185/cooking-equipment?pagesize=600")
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.entity-product-price-wrap.grid-item-price-wrap"))).click()
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "input#gv-postcalcode"))).send_keys("B3K 1X2")
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "a.gv-red-btn.gv-set-postal"))).click()
wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR, "a.gv-red-btn.gv-set-postal")))
time.sleep(3) # 无需等待60秒，弹窗消失后会话状态就已生效

# 用Selenium提取产品链接
product_eles = driver.find_elements(By.CSS_SELECTOR, "div.entity-product-image-wrap a")
productlinks = [ele.get_attribute('href') for ele in product_eles]

# 复用已有会话访问产品页提取信息
for link in productlinks:
    driver.get(link)
    wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.regPriceValue")))
    skunumber = driver.find_element(By.CSS_SELECTOR, "table.product-details-table").text
    pricing = driver.find_element(By.CSS_SELECTOR, "div.regPriceValue").text
    print(skunumber, pricing)

driver.quit()

方案2：导出Selenium Cookie给requests使用 #

如果要继续使用requests+BeautifulSoup的技术栈，只要把Selenium会话里的Cookie导出，后续所有requests请求都携带该Cookie即可：

# 在邮编提交完成的代码后新增Cookie导出逻辑
cookies = driver.get_cookies()
requests_cookies = {}
for cookie in cookies:
    requests_cookies[cookie['name']] = cookie['value']

# 之后所有requests.get请求都加上cookies参数
r = requests.get(url, headers=headers, cookies=requests_cookies)

额外优化建议 #

• 你原有代码里重复请求了两次分类页，还有一处请求漏传了headers参数，这类冗余代码可以直接清理
• 60秒等待没有必要，过长的等待反而可能触发会话超时，只要确认邮编提交弹窗消失即可进入后续抓取逻辑

内容的提问来源于stack exchange，提问作者scubadivingfool