使用Python的Selenium访问网站时导航下拉框空白且无法定位元素

• 目标：使用Python版Selenium实现自动化流程：访问Shoppers Stop官网，悬停导航菜单栏并点击下拉选项进入对应页面
• 异常现象：网站正常打开，但导航下拉框空白，点击下拉选项时触发「找不到元素」错误，下拉框无内容
• 已尝试：代码在Amazon.in测试下拉功能正常；在操作间添加time.sleep()，问题未解决

Message: no such element: Unable to locate element: {"method":"xpath","selector":"//a[text()='All Indian Wear']"}(Session info: chrome=123.0.6312.107)

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
import pandas as pd
import requests
import time
import re

# Set Chrome options
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36")

# Initialize driver with Chrome options
driver = webdriver.Chrome(options=chrome_options)

driver.get("https://www.shoppersstop.com/")     # Open the website
driver.maximize_window()        #Maximize window

try:
    WebDriverWait(driver, 20).until(lambda driver: driver.execute_script('return document.readyState') == 'complete')
    #time.sleep(10)
    actions = ActionChains(driver)
    women_category = driver.find_element(By.XPATH, "//html/body/main/header/nav/div[1]/div/ul[2]/li[4]/a")
    actions.move_to_element(women_category)          # Hover over the Women's category to reveal the dropdown
    actions.perform()
    #time.sleep(30)
    indian_wear = driver.find_element(By.XPATH, "//a[text()='All Indian Wear']")
    actions.move_to_element(indian_wear).perform().click()
    #time.sleep(20)
except Exception as e:
    print(e)

driver.quit()       # Close the browser

问题根源：

1. 仅等待document.readyState完成不足以确保动态加载的下拉菜单资源加载完毕
2. ActionChains链式调用错误，perform().click()无法正确触发点击动作
3. 旧版User-Agent可能被网站反爬机制识别，导致下拉菜单内容未渲染
4. 绝对路径XPath稳定性差，易因页面结构微小变动失效

修复步骤：

• 替换为当前Chrome版本匹配的User-Agent
• 添加Chrome选项禁用自动化特征识别
• 用显式等待等待下拉元素可被定位/可点击，而非固定sleep
• 修正ActionChains操作逻辑，先完成悬停，再等待元素出现后点击
• 使用更稳定的相对XPath定位元素

修改后的代码 #

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
import time

# 配置Chrome选项，规避反爬
chrome_options = webdriver.ChromeOptions()
# 替换为当前Chrome版本匹配的UA
chrome_options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.6312.107 Safari/537.36")
# 禁用自动化特征
chrome_options.add_argument("--disable-blink-features=AutomationControlled")
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', False)

driver = webdriver.Chrome(options=chrome_options)
driver.get("https://www.shoppersstop.com/")
driver.maximize_window()

try:
    # 等待页面核心导航元素加载，而非仅readyState
    WebDriverWait(driver, 20).until(
        EC.presence_of_element_located((By.XPATH, "//nav[contains(@class, 'header-nav')]"))
    )
    
    actions = ActionChains(driver)
    # 使用相对XPath定位Women分类，更稳定
    women_category = WebDriverWait(driver, 10).until(
        EC.element_to_be_clickable((By.XPATH, "//a[text()='Women']"))
    )
    # 执行悬停动作
    actions.move_to_element(women_category).perform()
    
    # 显式等待下拉选项加载完成并可点击
    indian_wear = WebDriverWait(driver, 15).until(
        EC.element_to_be_clickable((By.XPATH, "//a[text()='All Indian Wear']"))
    )
    # 直接点击目标元素
    indian_wear.click()
    
    # 可选：等待目标页面加载完成
    time.sleep(5)
except Exception as e:
    print(e)
finally:
    driver.quit()

关键改动说明 #

1. 反爬规避：添加禁用自动化特征的选项，更新UA为当前Chrome版本，避免网站识别为自动化脚本导致下拉菜单不渲染
2. 显式等待优化：用EC.element_to_be_clickable和EC.presence_of_element_located替代document.readyState，确保元素真正可交互
3. 元素定位优化：用文本匹配的相对XPath替代绝对路径，提升定位稳定性
4. ActionChains修正：悬停后直接等待目标元素可点击再执行点击，无需重复调用move_to_element，避免链式操作错误

内容的提问来源于stack exchange，提问作者Annie