Selenium+Python爬取资源网站遇NoSuchElementException问题求助

问题描述 #

尝试使用Selenium和Python爬取自有网站https://resources.norrag.org/categories/591,595中的案例研究标题、链接及摘要，因该网站为动态网站，放弃了Google Sheets方案，但运行代码时出现NoSuchElementException，无法定位指定元素。

原代码 #

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

s=Service('C:/Users/xxxx/Downloads/chromedriver_win32/chromedriver.exe')
browser = webdriver.Chrome(service=s)
url='https://resources.norrag.org/categories/591,595'
browser.get(url)

element = driver.find_element("xpath", '//div[@id="article_search_results"]//a')

print(element)
driver.close()

报错信息（中文翻译） #

---------------------------------------------------------------------------
NoSuchElementException                    Traceback (most recent call last)
Input In [8], in <cell line: 10>()
      6 url='https://resources.norrag.org/categories/591,595'
      7 driver.get(url)
---&gt; 10 element = driver.find_element("xpath", '//div[@id="article_search_results"]//a')
     12 print(element)
     13 driver.close()

File ~\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py:857, in WebDriver.find_element(self, by, value)
    854     by = By.CSS_SELECTOR
    855     value = '[name="%s"]' % value
--&gt; 857 return self.execute(Command.FIND_ELEMENT, {
    858     'using': by,
    859     'value': value})['value']

File ~\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py:435, in WebDriver.execute(self, driver_command, params)
    433 response = self.command_executor.execute(driver_command, params)
    434 if response:
--&gt; 435     self.error_handler.check_response(response)
    436     response['value'] = self._unwrap_value(
    437         response.get('value', None))
    438     return response

File ~\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py:247, in ErrorHandler.check_response(self, response)
    245         alert_text = value['alert'].get('text')
    246     raise exception_class(message, screen, stacktrace, alert_text)  # type: ignore[call-arg]  # mypy is not smart enough here
--&gt; 247 raise exception_class(message, screen, stacktrace)

NoSuchElementException: 消息：找不到元素：无法定位元素：{"method":"xpath","selector":"//div[@id="article_search_results"]//a"} (会话信息：chrome=103.0.5060.114)
堆栈跟踪：
回溯:
 Ordinal0 [0x00575FD3+2187219]
 Ordinal0 [0x0050E6D1+1763025]
 Ordinal0 [0x00423E78+802424]
 Ordinal0 [0x00451C10+990224]
 Ordinal0 [0x00451EAB+990891]
 Ordinal0 [0x0047EC92+1174674]
 Ordinal0 [0x0046CBD4+1100756]
 Ordinal0 [0x0047CFC2+1167298]
 Ordinal0 [0x0046C9A6+1100198]
 Ordinal0 [0x00446F80+946048]
 Ordinal0 [0x00447E76+949878]
 GetHandleVerifier [0x008190C2+2721218]
 GetHandleVerifier [0x0080AAF0+2662384]
 GetHandleVerifier [0x0060137A+526458]
 GetHandleVerifier [0x00600416+522518]
 Ordinal0 [0x00514EAB+1789611]
 Ordinal0 [0x005197A8+1808296]
 Ordinal0 [0x00519895+1808533]
 Ordinal0 [0x005226C1+1844929]
 BaseThreadInitThunk [0x76B5FA29+25]
 RtlGetAppContainerNamedObjectPath [0x77007A9E+286]
 RtlGetAppContainerNamedObjectPath [0x77007A6E+238]

解决方案 #

1. 修正变量名不一致问题 #

代码中初始化了browser = webdriver.Chrome(service=s)，但后续操作却使用未定义的driver变量，导致逻辑错误。需统一变量名，比如全部改为driver。

2. 添加显式等待处理动态加载 #

动态网站的元素需要时间渲染，直接查找元素会因元素未加载完成报错。使用WebDriverWait配合expected_conditions等待元素出现，避免因加载延迟导致的定位失败。

3. 验证元素定位表达式 #

通过浏览器开发者工具（F12）检查页面结构，确认是否存在id="article_search_results"的div元素，且内部包含目标a标签。若结构有变化，需调整XPath表达式。

修正后的代码示例 #

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

s = Service('C:/Users/xxxx/Downloads/chromedriver_win32/chromedriver.exe')
driver = webdriver.Chrome(service=s)
url = 'https://resources.norrag.org/categories/591,595'
driver.get(url)

# 等待元素加载，最长等待10秒
try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.XPATH, '//div[@id="article_search_results"]//a'))
    )
    print("标题:", element.text)
    print("链接:", element.get_attribute('href'))
finally:
    driver.close()

内容的提问来源于stack exchange，提问作者Paul Gerhard