使用Selenium无法运行Chrome headless模式的问题求助

问题背景 #

运行以下Selenium代码时，开启--headless参数会抛出错误，注释掉该参数则正常运行，推测目标网站对Headless模式做了检测拦截：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
import time
from fake_useragent import UserAgent
import random

ua = UserAgent()

options = Options()
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--blink-settings=imagesEnabled=false')
chrome_options.add_argument('--headless')
chrome_options.add_argument(f'user-agent={ua.random}')
driver = webdriver.Chrome(options=options, chrome_options=chrome_options)
driver.maximize_window()
url = "https://magiceden.io/marketplace/hasuki"
driver.get(url)
element = driver.find_element(By.CSS_SELECTOR, "#content > div.tw-w-full.tw-py-0.sm\:tw-mt-0 > div.tw-flex.tw-relative > div.tw-flex-auto.tw-max-w-full.tw-pt-0 > div.tw-flex.tw-items-center.md\:tw-justify-between.tw-gap-2.md\:tw-gap-4.md\:tw-sticky.tw-top-\[133px\].tw-bg-gray-100.tw-z-10.tw-flex-wrap.tw-p-5 > div.tw-flex.tw-flex-grow.tw-justify-center.tw-gap-x-2 > button > span:nth-child(4)")
print(f"The current instant sell price is {element.text}")

错误回溯信息：

Backtrace:
        (No symbol) [0x00806643]
        (No symbol) [0x0079BE21]
        (No symbol) [0x0069DA9D]
        (No symbol) [0x006D1342]
        (No symbol) [0x006D147B]
        (No symbol) [0x00708DC2]
        (No symbol) [0x006EFDC4]
        (No symbol) [0x00706B09]
        (No symbol) [0x006EFB76]
        (No symbol) [0x006C49C1]
        (No symbol) [0x006C5E5D]
        GetHandleVerifier [0x00A7A142+2497106]
        GetHandleVerifier [0x00AA85D3+2686691]
        GetHandleVerifier [0x00AABB9C+2700460]
        GetHandleVerifier [0x008B3B10+635936]
        (No symbol) [0x007A4A1F]
        (No symbol) [0x007AA418]
        (No symbol) [0x007AA505]
        (No symbol) [0x007B508B]
        BaseThreadInitThunk [0x75EB00F9+25]
        RtlGetAppContainerNamedObjectPath [0x77A27BBE+286]
        RtlGetAppContainerNamedObjectPath [0x77A27B8E+238]

解决方法 #

目标网站通常通过浏览器特征识别Headless模式，添加以下参数模拟真实浏览器环境即可绕过检测：

• 使用新版Headless模式：--headless=new（Chrome 112+版本推荐，行为更接近正常浏览器）
• 禁用自动化检测：--disable-blink-features=AutomationControlled
• 设置标准窗口尺寸：--window-size=1920,1080（避免Headless默认小窗口被识别）
• 统一Options对象：原代码中同时使用Options()和ChromeOptions()属于冗余写法，新版Selenium只需用ChromeOptions()

修正后的完整代码 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

ua = UserAgent()

chrome_options = Options()
# 禁用图片加载提升速度
chrome_options.add_argument('--blink-settings=imagesEnabled=false')
# 新版Headless模式
chrome_options.add_argument('--headless=new')
# 随机User-Agent
chrome_options.add_argument(f'user-agent={ua.random}')
# 禁用自动化检测特征
chrome_options.add_argument('--disable-blink-features=AutomationControlled')
# 设置窗口尺寸
chrome_options.add_argument('--window-size=1920,1080')

driver = webdriver.Chrome(options=chrome_options)
url = "https://magiceden.io/marketplace/hasuki"
driver.get(url)

try:
    # 显式等待元素可见，最长等待10秒，避免页面未加载完成导致查找失败
    element = WebDriverWait(driver, 10).until(
        EC.visibility_of_element_located((By.CSS_SELECTOR, "#content > div.tw-w-full.tw-py-0.sm\:tw-mt-0 > div.tw-flex.tw-relative > div.tw-flex-auto.tw-max-w-full.tw-pt-0 > div.tw-flex.tw-items-center.md\:tw-justify-between.tw-gap-2.md\:tw-gap-4.md\:tw-sticky.tw-top-\[133px\].tw-bg-gray-100.tw-z-10.tw-flex-wrap.tw-p-5 > div.tw-flex.tw-flex-grow.tw-justify-center.tw-gap-x-2 > button > span:nth-child(4)"))
    print(f"The current instant sell price is {element.text}")
finally:
    # 确保浏览器关闭
    driver.quit()

额外说明 #

• Headless模式下driver.maximize_window()可能无效，用--window-size参数设置尺寸更可靠
• 替换原代码的即时元素查找为显式等待，能有效避免因页面加载延迟导致的元素未找到问题

内容的提问来源于stack exchange，提问作者syrok