使用Selenium爬取GPAHE仇恨符号库遇NoSuchElementException问题求助

核心问题分析 #

你遇到的NoSuchElementException主要由三个原因导致：

1. 绝对XPATH过于脆弱：你使用的绝对路径完全依赖DOM层级，页面结构哪怕有微小变化都会直接失效，容错性极低。
2. 未处理网站反爬检测：该网站可能识别出了Selenium的自动化特征（比如ChromeDriver的默认标识），导致元素加载异常或被屏蔽。
3. 等待机制不合理：time.sleep是固定等待，无法确保元素真正加载完成；之前的显式等待可能未正确等待元素的可交互状态。

另外，循环点击Load More的逻辑过于粗糙，直接用find_element捕获异常容易误判，且未处理元素陈旧（StaleElementReferenceException）的情况。

修复方案 #

1. 替换为稳定的相对定位 #

放弃绝对XPATH，改用基于元素属性或父元素特征的相对定位：

• Symbol类型输入框：通过placeholder属性定位 //input[@placeholder='Filter by Symbol Type']
• Load More按钮：通过文本内容定位 //button[contains(text(), 'Load More')]

2. 隐藏Selenium自动化特征 #

给ChromeDriver添加启动参数，模拟真实浏览器环境：

options = webdriver.ChromeOptions()
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument('--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options)
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")

3. 正确使用显式等待 #

用WebDriverWait等待元素可交互而非仅存在，确保元素能被正常操作：

wait = WebDriverWait(driver, 20)
input_element = wait.until(EC.element_to_be_clickable((By.XPATH, "//input[@placeholder='Filter by Symbol Type']")))

4. 优化Load More点击逻辑 #

每次点击前重新等待元素，避免陈旧元素异常：

while True:
    try:
        load_more_btn = wait.until(EC.element_to_be_clickable((By.XPATH, "//button[contains(text(), 'Load More')]")))
        driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", load_more_btn)
        load_more_btn.click()
        wait.until(EC.staleness_of(load_more_btn))  # 等待按钮状态更新
        time.sleep(1)
    except:
        break

修改后的完整代码 #

import time
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException, StaleElementReferenceException
from selenium import webdriver
import json

def parse_profile(driver, hatesymbol_url):
    image_list = []
    try:
        driver.get(hatesymbol_url)
        wait = WebDriverWait(driver, 15)
        # 等待图片加载完成
        wait.until(EC.presence_of_element_located((By.XPATH, "//div[@class='css-1l7c0cy']//img")))
        
        images = driver.find_elements(By.XPATH, "//div[@class='css-1l7c0cy']//img[@class='MuiCardMedia-root MuiCardMedia-media MuiCardMedia-img paper css-rhsghg']")
        for image in images:
            image_url = image.get_attribute("src")
            if image_url:
                image_list.append(image_url)
    except (StaleElementReferenceException, NoSuchElementException) as e:
        print(f"解析页面出错 {hatesymbol_url}: {str(e)}")
    return image_list

# 配置Chrome选项，隐藏自动化特征
options = webdriver.ChromeOptions()
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_argument('--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)

driver = webdriver.Chrome(options=options)
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.maximize_window()

gpahe_url = "https://globalextremism.org/global-extremist-symbols-database/"
driver.get(gpahe_url)
wait = WebDriverWait(driver, 20)

symbol_types = ['Hand Sign', 'Hate Image', 'Hateful Clothing', 'Hateful Literature', 'Hateful Media', 'Hateful Music', "Text"]
all_symbols = []

# 等待输入框加载并定位
input_element = wait.until(EC.element_to_be_clickable((By.XPATH, "//input[@placeholder='Filter by Symbol Type']")))

for symbol_type in symbol_types:
    # 清空输入框再输入新类型，避免内容叠加
    input_element.clear()
    input_element.send_keys(symbol_type)
    time.sleep(1)  # 等待筛选结果加载
    
    # 循环点击Load More加载全部内容
    while True:
        try:
            load_more_btn = wait.until(EC.element_to_be_clickable((By.XPATH, "//button[contains(text(), 'Load More')]")))
            driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", load_more_btn)
            load_more_btn.click()
            wait.until(EC.staleness_of(load_more_btn))
            time.sleep(1)
        except:
            print(f"{symbol_type}类型已加载全部内容")
            break
    
    # 获取当前类型的所有符号元素
    elems = wait.until(EC.presence_of_all_elements_located((By.XPATH, "//div[@class='horizontal-list-item MuiBox-root css-zdcb3j']")))
    for elem in elems:
        symbol = {}
        try:
            # 提取名称
            name_elem = elem.find_element(By.XPATH, ".//h2[contains(@class, 'css-vi1jz3')]")
            symbol['name'] = name_elem.text.strip()
            
            # 提取详情URL
            url_elem = elem.find_element(By.XPATH, ".//a[contains(@class, 'css-n8dvpa')]")
            symbol['metadata'] = {'url': url_elem.get_attribute("href"), 'type': symbol_type}
            
            # 提取意识形态标签
            ideo_elems = elem.find_elements(By.XPATH, ".//div[@class='css-zq5ip9']//span[@class='MuiChip-label MuiChip-labelMedium css-1cjkvaf']")
            symbol['metadata']['ideology'] = [ideo.text.strip() for ideo in ideo_elems]
            
            # 提取位置标签
            loc_elems = elem.find_elements(By.XPATH, ".//div[@class='css-yv67l3']//span[@class='MuiChip-label MuiChip-labelMedium css-1cjkvaf']")
            symbol['metadata']['location'] = [loc.text.strip() for loc in loc_elems]
            
            # 提取描述内容
            desc_elem = elem.find_element(By.XPATH, ".//div[contains(@class, 'css-11b53ld')]")
            symbol['metadata']['description'] = desc_elem.text.strip()
            
            all_symbols.append(symbol)
        except Exception as e:
            print(f"解析符号元素出错: {str(e)}")
            continue

# 爬取每个符号的图片信息
for symbol in all_symbols:
    if 'url' in symbol['metadata']:
        symbol_images = parse_profile(driver, symbol['metadata']['url'])
        symbol['official_image'] = symbol_images

# 保存结果到JSON文件
with open('hate_symbols.json', 'w', encoding='utf-8') as f:
    json.dump(all_symbols, f, ensure_ascii=False, indent=4)

driver.quit()

额外注意事项 #

• 网站可能更新反爬策略，若再次出现定位失败，需检查元素属性是否变化并调整定位方式。
• 适当控制请求频率，避免过快操作导致IP被临时封禁。
• 爬取前请确认网站使用条款，确保行为合规。

内容的提问来源于stack exchange，提问作者Noey