Selenium(Python)被网站拦截，如何绕过反爬获取数据？

问题 #

需要用Selenium访问以下两个股票网站获取ROE数据，存入roe_list：

• https://www.valueresearchonline.com/stocks/44052/reliance-industries-ltd
• https://www.valueresearchonline.com/stocks/44811/tata-consultancy-services-ltd

使用代码如下：

def fetch_roe(link):
    URL = link
    browser.get(URL)
    browser.maximize_window()
    time.sleep(5)
    
    roe = browser.find_element('xpath', '/html/body/section[2]/div/div/div[1]/div/div[4]/section[1]/div[1]/div/div[2]/div[2]/div/div/div[2]/table/tbody/tr[2]/td[2]/div')

    return roe.text


roe_list = []

option = Options()
option.add_argument("start-maximized")
option.binary_location = brave_path
browser = webdriver.Chrome(executable_path=driver_path, options=option)

for url_link in vro_list:
    print(url_link)
    roe_item = fetch_roe(url_link)
    roe_list.append(roe_item)
    time.sleep(5)

browser.quit()
    
print(roe_list)

运行时第二次迭代触发NoSuchElementException，检查发现网站拦截了WebDriver，页面显示"Brave is being controlled by automated software"，完整错误栈如下：

NoSuchElementException                    Traceback (most recent call last)
Input In [29], in <cell line: 8>()
      8 for url_link in vro_list:
      9     print(url_link)
---> 10     roe_item = fetch_roe(url_link)
     11     roe_list.append(roe_item)
     12     time.sleep(5)

Input In [27], in fetch_roe(link)
      6 time.sleep(5)
      8 #name = browser.find_element('xpath', '/html/body/div[3]/h1/span')
----> 9 roe = browser.find_element('xpath', '/html/body/section[2]/div/div/div[1]/div/div[4]/section[1]/div[1]/div/div[2]/div[2]/div/div/div[2]/table/tbody/tr[2]/td[2]/div')
     11 return roe.text

File ~\anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py:855, in WebDriver.find_element(self, by, value)
    852     by = By.CSS_SELECTOR
    853     value = '[name="%s"]' % value
--> 855 return self.execute(Command.FIND_ELEMENT, {
    856     'using': by,
    857     'value': value})['value']

File ~\anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py:428, in WebDriver.execute(self, driver_command, params)
    426 response = self.command_executor.execute(driver_command, params)
    427 if response:
--> 428     self.error_handler.check_response(response)
    429     response['value'] = self._unwrap_value(
    430         response.get('value', None))
    431     return response

File ~\anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py:243, in ErrorHandler.check_response(self, response)
    241         alert_text = value['alert'].get('text')
    242     raise exception_class(message, screen, stacktrace, alert_text)  # type: ignore[call-arg]  # mypy is not smart enough here
--> 243 raise exception_class(message, screen, stacktrace)

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/section[2]/div/div/div[1]/div/div[4]/section[1]/div[1]/div/div[2]/div[2]/div/div/div[2]/table/tbody/tr[2]/td[2]/div"}
  (Session info: chrome=105.0.5195.102)
Stacktrace:
Backtrace:
    Ordinal0 [0x00A2C0A3+2212003]
    Ordinal0 [0x009C2CC1+1780929]
    Ordinal0 [0x008D465D+804445]
    Ordinal0 [0x00903475+996469]
    Ordinal0 [0x0090363B+996923]
    Ordinal0 [0x00931382+1184642]
    Ordinal0 [0x0091EC64+1109092]
    Ordinal0 [0x0092F5B2+1177010]
    Ordinal0 [0x0091EA36+1108534]
    Ordinal0 [0x008F83C9+951241]
    Ordinal0 [0x008F9396+955286]
    GetHandleVerifier [0x00CD9CE2+2746722]
    GetHandleVerifier [0x00CCA234+2682548]
    GetHandleVerifier [0x00ABB34A+524234]
    GetHandleVerifier [0x00AB9B60+518112]
    Ordinal0 [0x009C9FBC+1810364]
    Ordinal0 [0x009CEA28+1829416]
    Ordinal0 [0x009CEB15+1829653]
    Ordinal0 [0x009D8744+1869636]
    BaseThreadInitThunk [0x76A4FA29+25]
    RtlGetAppContainerNamedObjectPath [0x77C07A9E+286]
    RtlGetAppContainerNamedObjectPath [0x77C07A6E+238]

解决方案 #

1. 隐藏WebDriver自动化特征 #

给ChromeOptions添加核心参数，直接禁用网站的自动化检测标识：

option = Options()
option.add_argument("start-maximized")
option.binary_location = brave_path
# 核心：禁用自动化控制提示
option.add_argument('--disable-blink-features=AutomationControlled')
# 排除自动化扩展相关提示
option.add_experimental_option("excludeSwitches", ["enable-automation"])
option.add_experimental_option('useAutomationExtension', False)
# 辅助参数：减少被检测概率
option.add_argument('--no-sandbox')
option.add_argument('--disable-dev-shm-usage')
option.add_argument('--disable-gpu')

2. 注入stealth脚本增强伪装 #

通过Chrome DevTools协议注入stealth.min.js（可直接下载该脚本到本地），彻底隐藏WebDriver的底层特征：

browser = webdriver.Chrome(executable_path=driver_path, options=option)
# 注入stealth脚本
with open('stealth.min.js', 'r') as f:
    stealth_script = f.read()
browser.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
    "source": stealth_script
})

3. 替换绝对XPath为相对定位 #

绝对XPath极易受页面结构变化影响，改用基于文本的相对XPath定位ROE元素，稳定性更强：

# 根据"Return on Equity"文本定位对应行的第二个单元格
roe = browser.find_element('xpath', '//table/tbody/tr[td[text()="Return on Equity"]]/td[2]/div')

4. 用智能等待替代固定sleep #

固定sleep不仅低效还容易触发反爬，改用WebDriverWait等待元素加载完成，更贴合正常用户行为：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

def fetch_roe(link, browser):
    browser.get(link)
    # 最长等待10秒，直到ROE元素出现
    roe = WebDriverWait(browser, 10).until(
        EC.presence_of_element_located(('xpath', '//table/tbody/tr[td[text()="Return on Equity"]]/td[2]/div'))
    )
    return roe.text

完整修改后代码 #

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

def fetch_roe(link, browser):
    browser.get(link)
    # 智能等待ROE元素加载
    roe = WebDriverWait(browser, 10).until(
        EC.presence_of_element_located(('xpath', '//table/tbody/tr[td[text()="Return on Equity"]]/td[2]/div'))
    )
    return roe.text


roe_list = []
vro_list = [
    "https://www.valueresearchonline.com/stocks/44052/reliance-industries-ltd",
    "https://www.valueresearchonline.com/stocks/44811/tata-consultancy-services-ltd"
]

option = Options()
option.add_argument("start-maximized")
option.binary_location = brave_path  # 替换为你的Brave浏览器路径
# 隐藏自动化特征
option.add_argument('--disable-blink-features=AutomationControlled')
option.add_experimental_option("excludeSwitches", ["enable-automation"])
option.add_experimental_option('useAutomationExtension', False)
option.add_argument('--no-sandbox')
option.add_argument('--disable-dev-shm-usage')

browser = webdriver.Chrome(executable_path=driver_path, options=option)  # 替换为你的ChromeDriver路径

# 注入stealth脚本（需提前下载stealth.min.js到本地目录）
with open('stealth.min.js', 'r') as f:
    stealth_script = f.read()
browser.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
    "source": stealth_script
})

for url_link in vro_list:
    print(url_link)
    roe_item = fetch_roe(url_link, browser)
    roe_list.append(roe_item)
    time.sleep(2)  # 缩短等待时间，模拟正常浏览间隔

browser.quit()
    
print(roe_list)

内容的提问来源于stack exchange，提问作者Deepanshu Bansal