网站爬取异常：无法获取Sephora护肤品商品URL的技术求助

问题背景 #

需要爬取Sephora网站护肤品板块的商品URL，后续将获取更多相关数据，但当前卡在URL获取环节。使用的Python模块如下：

import bs4
import pandas as pd
import numpy as np
import random
import requests
from lxml import etree
import time
from tqdm.notebook import tqdm

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import ElementNotInteractableException
from time import sleep
from webdriver_manager.chrome import ChromeDriverManager

尝试的三种无效方法 #

1. 复杂XPATH定位（结果urls列表为空） #

driver = webdriver.Chrome(ChromeDriverManager().install())
urls = []

for page in tqdm(range(2, 10)):
    driver.get("https://www.sephora.com/shop/skincare?currentPage="+str(page))
    for order in tqdm(range(0,70)):
        skincare = driver.find_elements(By.XPATH, "//main[@class='css-1owb2na']//div[@data-comp='ProductGrid ']//div[@class='css-1322gsb']//div[@class='css-1qe8tjm'][@style='order:" +str(order)+ " ;']//a[@class='css-klx76']")
        for _skincare in skincare:
            urls.append({"url":_skincare.get_attribute('href')})
driver.quit()

2. 简化XPATH定位（结果同样为空） #

driver = webdriver.Chrome(ChromeDriverManager().install())
urls = []

for page in tqdm(range(2, 10)):
    driver.get("https://www.sephora.com/shop/skincare?currentPage="+str(page))
    for order in tqdm(range(0,70)):
        skincare = driver.find_elements(By.XPATH, "//div[@style='order:" +str(order)+ " ;']//a[@class='css-klx76']")
        for _skincare in skincare:
            urls.append({"url":_skincare.get_attribute('href')})
driver.quit()

3. CLASS_NAME定位（仅获取少量URL） #

driver = webdriver.Chrome(ChromeDriverManager().install())
urls = []
for page in tqdm(range(2, 50)):
    driver.get("https://www.sephora.com/shop/skincare?currentPage="+str(page))
    skincare = driver.find_elements(By.CLASS_NAME, 'css-klx76')
    for _skincare in skincare:
        urls.append({"url":_skincare.get_attribute('href')})
driver.quit()

问题根源与解决方案 #

核心问题分析 #

• 动态CSS类名：css-klx76、css-1qe8tjm这类带随机后缀的类名是网站动态生成的，页面更新或不同环境下会失效，不能作为稳定定位依据
• 未等待页面加载：直接执行元素查找时，商品元素还未渲染完成，导致返回空列表
• 依赖不稳定属性：order是布局用的动态属性，并非商品元素的固定标识，不能用来定位

可行解决方案 #

1. 使用稳定语义化定位+显式等待

优先选择包含商品特征的属性（如/product/链接）和语义化类名，配合显式等待确保元素加载完成：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager
from tqdm.notebook import tqdm
import time
import random

# 配置浏览器模拟真实环境
options = webdriver.ChromeOptions()
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")

driver = webdriver.Chrome(ChromeDriverManager().install(), options=options)
driver.implicitly_wait(8)  # 全局隐式等待
urls = []

for page in tqdm(range(1, 10)):
    page_url = f"https://www.sephora.com/shop/skincare?currentPage={page}"
    driver.get(page_url)
    
    # 显式等待商品链接加载
    try:
        WebDriverWait(driver, 15).until(
            EC.presence_of_element_located((By.XPATH, "//a[contains(@href, '/product/') and @data-at='product_item_name']"))
        )
    except Exception as e:
        print(f"第{page}页加载超时：{str(e)}")
        continue
    
    # 滚动页面加载所有商品（Sephora滚动加载未显示的商品）
    last_scroll_height = driver.execute_script("return document.body.scrollHeight")
    while True:
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(random.uniform(1.5, 3))
        new_scroll_height = driver.execute_script("return document.body.scrollHeight")
        if new_scroll_height == last_scroll_height:
            break
        last_scroll_height = new_scroll_height
    
    # 获取所有商品链接并去重
    product_links = driver.find_elements(By.XPATH, "//a[contains(@href, '/product/') and @data-at='product_item_name']")
    for link in product_links:
        href = link.get_attribute("href")
        if href not in [item["url"] for item in urls]:
            urls.append({"url": href})
    
    print(f"第{page}页成功获取{len(product_links)}个商品URL")

driver.quit()
print(f"总计获取到{len(urls)}个唯一商品URL")

2. 避免动态CSS类，使用相对路径

改用语义化的类名前缀定位，比如：

//div[contains(@class, 'ProductGrid')]//div[contains(@class, 'ProductCard')]//a[contains(@href, '/product/')]

这类定位不依赖完整的动态类名，仅匹配语义化前缀，稳定性更强。

3. 优化反爬应对

• 添加随机等待时间，避免请求频率过高
• 配置浏览器UA和禁用自动化检测，模拟真实用户行为
• 可考虑使用代理IP分散请求来源（若遇到IP封禁）

内容的提问来源于stack exchange，提问作者user20576555