Selenium无法打开有效URL的whois EURID网站，仅显示空白页

问题描述 #

使用Selenium启动Chrome/Firefox浏览器访问EURID Whois站点（https://whois.eurid.eu/en/search/?domain=xxxx.eu）时，页面显示空白，此前访问正常，推测网站启用了反爬机制识别自动化工具。

当前驱动配置代码：

def set_driver() -> WebDriver:
    service = Service()
    if cfg["browser"]["browser_name"].strip().lower() == "firefox":
        options = webdriver.FirefoxOptions()
        driver = webdriver.Firefox(service=service, options=options)
    elif cfg["browser"]["browser_name"].strip().lower() == "edge":
        options = webdriver.EdgeOptions()
        driver = webdriver.Edge(service=service, options=options)
    else:
        options = webdriver.ChromeOptions()
        driver = webdriver.Chrome(service=service, options=options)
    return driver

访问代码：

domain_name = "ddddddd.eu"  # any name ending with '.eu'
URL = f"https://whois.eurid.eu/en/search/?domain={domain_name}"
drv = set_driver()
drv.get(URL) 
time.sleep(5)

解决方案 #

1. 添加浏览器伪装参数，规避自动化识别 #

网站通常通过检测浏览器的自动化特征（如webdriver标识、特定配置）拦截Selenium请求，需给驱动添加参数伪装成真实浏览器：

Chrome配置修改

options = webdriver.ChromeOptions()
# 禁用自动化控制标识
options.add_argument("--disable-blink-features=AutomationControlled")
# 隐藏Chrome正被自动化控制的提示
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
# 设置真实用户代理（可替换为当前浏览器的UA）
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")

Firefox配置修改

options = webdriver.FirefoxOptions()
# 禁用webdriver标识
options.set_preference("dom.webdriver.enabled", False)
options.set_preference('useAutomationExtension', False)
# 禁用自动化提示
options.add_argument("--disable-infobars")
# 设置真实用户代理
options.set_preference("general.useragent.override", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/118.0")

2. 更新驱动配置函数 #

整合上述参数后的完整驱动配置：

def set_driver() -> WebDriver:
    service = Service()
    browser_name = cfg["browser"]["browser_name"].strip().lower()
    
    if browser_name == "firefox":
        options = webdriver.FirefoxOptions()
        options.set_preference("dom.webdriver.enabled", False)
        options.set_preference('useAutomationExtension', False)
        options.add_argument("--disable-infobars")
        options.set_preference("general.useragent.override", "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/118.0")
        driver = webdriver.Firefox(service=service, options=options)
    elif browser_name == "edge":
        options = webdriver.EdgeOptions()
        options.add_argument("--disable-blink-features=AutomationControlled")
        options.add_experimental_option("excludeSwitches", ["enable-automation"])
        options.add_experimental_option('useAutomationExtension', False)
        options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Edge/118.0.2088.61 Safari/537.36")
        driver = webdriver.Edge(service=service, options=options)
    else:
        options = webdriver.ChromeOptions()
        options.add_argument("--disable-blink-features=AutomationControlled")
        options.add_experimental_option("excludeSwitches", ["enable-automation"])
        options.add_experimental_option('useAutomationExtension', False)
        options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")
        driver = webdriver.Chrome(service=service, options=options)
    
    # 执行脚本进一步隐藏自动化特征
    driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
    return driver

3. 优化页面等待策略 #

替换time.sleep(5)为显式等待，确保页面元素加载完成后再进行操作：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException

domain_name = "ddddddd.eu"
URL = f"https://whois.eurid.eu/en/search/?domain={domain_name}"
drv = set_driver()
drv.get(URL)

# 等待页面核心元素加载，比如搜索结果区域
try:
    WebDriverWait(drv, 10).until(
        EC.presence_of_element_located((By.CSS_SELECTOR, ".whois-result"))
    )
    # 后续抓取逻辑
except TimeoutException:
    print("页面加载超时，仍为空白页")

4. 额外排查点 #

• 确认浏览器版本与WebDriver版本完全匹配（如Chrome 118对应ChromeDriver 118）
• 手动访问目标URL，确认网络环境正常、网站可正常打开
• 若仍无效，可尝试添加代理IP、禁用JavaScript（部分反爬依赖JS检测）

内容的提问来源于stack exchange，提问作者CC PL