Selenium加载动态页面后脚本冻结，请求技术支持

可能的原因及对应修复方案 #

1. ChromeDriver与Chrome版本不兼容 #

这是最常见的问题，手动指定的ChromeDriver版本必须和本地安装的Chrome版本完全匹配。推荐用webdriver-manager自动管理驱动，避免版本不匹配问题：

• 先安装依赖：

  pip install webdriver-manager
  

• 修改驱动初始化代码：

  from selenium.webdriver.chrome.service import Service
  from webdriver_manager.chrome import ChromeDriverManager
  
  # 替换原有的driver初始化行
  driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)
  

2. 页面存在未处理的Cookie弹窗 #

页面加载后弹出的Cookie授权框可能导致脚本卡住，需要先处理弹窗：

driver.get("https://weighttraining.guide/exercises/standing-dumbbell-overhead-triceps-extension/")
print("Selenium get successful")

# 尝试点击Cookie同意按钮
try:
    cookie_btn = WebDriverWait(driver, 5).until(
        EC.element_to_be_clickable((By.XPATH, "//button[contains(text(), 'Accept')]"))
    )
    cookie_btn.click()
except:
    # 无弹窗则跳过
    pass

3. 等待条件选择不当 #

presence_of_element_located仅判断元素存在，不保证元素可见或渲染完成，换成visibility_of_element_located更可靠：

WebDriverWait(driver, 10).until(
    EC.visibility_of_element_located((By.CLASS_NAME, "entry-content"))
)

4. 禁用Chrome自动化检测 #

部分网站会识别Selenium的自动化特征，导致页面加载异常，添加以下选项规避检测：

chrome_options.add_argument("--disable-blink-features=AutomationControlled")
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', False)

5. 启用无头模式（可选） #

如果不需要可视化浏览器，启用无头模式可避免交互类问题：

chrome_options.add_argument("--headless=new")

完整修复后的示例代码 #

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

user_agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(f"user-agent={user_agent}")
# 禁用自动化检测
chrome_options.add_argument("--disable-blink-features=AutomationControlled")
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', False)
# 可选：启用无头模式
# chrome_options.add_argument("--headless=new")

try:
    driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)
    print("Driver loaded")
    driver.get("https://weighttraining.guide/exercises/standing-dumbbell-overhead-triceps-extension/")
    print("Selenium get successful")

    # 处理Cookie弹窗
    try:
        cookie_btn = WebDriverWait(driver, 5).until(
            EC.element_to_be_clickable((By.XPATH, "//button[contains(text(), 'Accept')]"))
        )
        cookie_btn.click()
    except Exception as cookie_e:
        print(f"No cookie popup or failed to click: {cookie_e}")

    # 等待目标元素可见
    WebDriverWait(driver, 10).until(
        EC.visibility_of_element_located((By.CLASS_NAME, "entry-content"))
    )
    print("Page loaded")
    # 此处可添加Beautiful Soup解析逻辑
    # page_source = driver.page_source
    # soup = BeautifulSoup(page_source, 'html.parser')
    driver.quit()
except Exception as e:
    print(f"An error occurred: {e}")
    # 确保出错时关闭驱动
    if 'driver' in locals():
        driver.quit()

内容的提问来源于stack exchange，提问作者BLimitless