基于Selenium的Python爬虫无法定位元素：如何抓取商品名与价格？

问题现象 #

编写基于Selenium的Python网页爬虫时，持续抛出no such element: unable to locate element错误，但在Selenium启动的浏览器中可以看到目标元素。

目标页面 #

https://www.neimanmarcus.com/p/givenchy-g-chain-ring-prod250190244?childItemId=NMY5X1R_&navpath=cat000000_cat4870731_cat50910737_cat2650734&page=0&position=11

Driver初始化代码 #

def getDriver():
    try:
        options = webdriver.ChromeOptions()
        options.add_argument("start-maximized")
        options.add_experimental_option("excludeSwitches", ["enable-automation"])
        options.add_experimental_option('useAutomationExtension', False)
        options.add_experimental_option('excludeSwitches', ['enable-logging'])
        driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=options)
        return driver
    except Exception:
        traceback.print_exc()
        print(Exception)

核心爬虫代码 #

def getProduct(domain, url):
    # find the yaml file for the domain
    driver = getDriver()

    noYaml = False
    try:
        markupPath = 'markups/%s.yml' % domain
    except Exception as error:
        noYaml = True
    if(noYaml == False):
        with open(markupPath, 'r') as file:
            yamlElements = yaml.safe_load(file)

        titleXpath = '//span[contains(concat(" ",normalize-space(@class)," ")," Titlestyles__ProductName-fRyAwr ")]'
        priceXpath = '//span[contains(concat(" ",normalize-space(@class)," ")," Pricingstyles__RetailPrice-eYMMwV ")]'
        print("XXXXXXPATHHHHHHHHHHHHHS: ",titleXpath, priceXpath)
        driver.get(url)
        driver.implicitly_wait(10)
        
        try:
            # WebDriverWait(driver, 2).until(EC.presence_of_element_located((By.XPATH,titleXpath)))
            
            title = driver.find_element(By.XPATH, titleXpath)
        except Exception as error:
            print("ERROR: ", error)
            title = None

        try: 
            # WebDriverWait(driver, 2).until(EC.presence_of_element_located((By.XPATH,priceXpath)))
            price = driver.find_element_by_xpath(priceXpath)
            # price = driver.find_element(By.XPATH,priceXpath)
            print(price)
        except Exception as error:
            print("ERRROR: ", error)
            price = None
        driver.execute_script("window.stop();")
        data={}
        if(title != None):
            print(title.get_attribute('innerHTML'))
            data['title'] = title.get_attribute('innerHTML')

        if(price != None):
            print(price.get_attribute('innerHTML'))
            data['price'] = price.get_attribute('innerHTML')

        if 'price' not in data or data['price'] == "" or data['price'] == None:
            driver.quit()
            return False
            getProduct(domain, url)
        driver.quit()
        return data

具体错误信息 #

Message: no such element: Unable to locate element: {"method":"xpath","selector":"//span[contains(concat(" ",normalize-space(@class)," ")," Pricingstyles__RetailPrice-eYMMwV ")]"}

问题原因与解决办法 #

1. 动态类名导致定位失效 #

网站使用CSS Modules类的生成技术，类名（如Pricingstyles__RetailPrice-eYMMwV）为动态可变值，每次页面加载都会变化，直接用完整类名定位必然失效。

解决方式：

• 改用更稳定的定位逻辑，比如匹配类名前缀、元素文本、固定属性（如data-testid）或层级关系：

  # 匹配包含RetailPrice的类名前缀
  priceXpath = '//span[contains(normalize-space(@class), "RetailPrice")]'
  # 或用CSS选择器
  price = driver.find_element(By.CSS_SELECTOR, "span[class*='RetailPrice']")
  

2. 等待策略不合理 #

仅依赖implicitly_wait(10)的隐式等待对动态渲染元素支持不稳定，且原代码注释了显式等待逻辑，可能在元素未完全渲染时就尝试定位。

解决方式：

• 使用显式等待（WebDriverWait）配合visibility_of_element_located条件（确保元素可见），替换原定位逻辑：

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.support import expected_conditions as EC
  
  driver.get(url)
  
  # 显式等待标题可见，超时15秒
  try:
      title = WebDriverWait(driver, 15).until(
          EC.visibility_of_element_located((By.XPATH, '//span[contains(normalize-space(@class), "ProductName")]'))
      )
  except Exception as error:
      print("ERROR: ", error)
      title = None
  
  # 显式等待价格可见
  try:
      price = WebDriverWait(driver, 15).until(
          EC.visibility_of_element_located((By.XPATH, '//span[contains(normalize-space(@class), "RetailPrice")]'))
      )
  except Exception as error:
      print("ERRROR: ", error)
      price = None
  

3. 页面加载中断问题 #

原代码中driver.execute_script("window.stop();")会强制终止页面加载，可能导致目标元素未完全加载就被中断，建议删除该语句，让页面自然加载完成。

4. 重试逻辑无效 #

原代码中return False后的getProduct(domain, url)不会执行（return会直接终止函数），若需要重试需调整逻辑：

if 'price' not in data or data['price'] == "" or data['price'] == None:
    driver.quit()
    return getProduct(domain, url)

内容的提问来源于stack exchange，提问作者Tadewos Bellete