使用BeautifulSoup获取CoinMarketCap币种名称遇NoneType错误的解决方法

错误原因 #

• 部分<tr>元素中不存在你指定的p.sc-1eb5slv-0.iworPT标签，导致find()返回None，调用.text时触发NoneType错误。CoinMarketCap的动态类名（如sc-1eb5slv-0、iworPT）可能随时变更，或部分行是分隔线、空行，不包含目标元素。
• 无头模式下浏览器特征易被反爬机制识别，可能导致页面未完整渲染目标元素，即使你认为页面已加载完成。

修复方案 #

1. 增加空值判断，避免报错 #

在访问.text前先检查元素是否存在，跳过无目标元素的行：

for crypto_details in crypto_detail:
    crypto_name = crypto_details.find("p", class_="sc-1eb5slv-0 iworPT")
    if crypto_name:  # 仅当元素存在时执行后续操作
        print(crypto_name.text.strip())
    sleep(randint(2, 10))

2. 使用更稳定的选择器 #

替换动态生成的类名，改用更持久的元素结构或属性定位。例如，通过加密货币的链接标签（cmc-link）定位：

for crypto_details in crypto_detail:
    crypto_link = crypto_details.find("a", class_="cmc-link")
    if crypto_link:
        crypto_name = crypto_link.find("p")
        if crypto_name:
            print(crypto_name.text.strip())
    sleep(randint(2, 10))

3. 优化页面加载逻辑 #

• 添加显式等待，确保目标元素加载完成后再解析：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

# 访问页面后添加等待
wait = WebDriverWait(driver, 15)
wait.until(EC.presence_of_all_elements_located((By.TAG_NAME, "tr")))

soup = BeautifulSoup(driver.page_source, "html.parser")

• 为无头模式添加真实浏览器的User-Agent，避免被反爬拦截：

chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")

完整修复后的代码示例 #

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from bs4 import BeautifulSoup
from time import sleep
from random import randint

chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")

URL = "https://coinmarketcap.com/"
driver = webdriver.Chrome(
    service=Service(ChromeDriverManager().install()), options=chrome_options
)
driver.get(URL)

# 等待表格行加载完成
wait = WebDriverWait(driver, 15)
wait.until(EC.presence_of_all_elements_located((By.TAG_NAME, "tr")))

soup = BeautifulSoup(driver.page_source, "html.parser")
results = soup.find("div", class_="grid")
crypto_detail = results.find_all("tr")

for crypto_details in crypto_detail:
    # 使用稳定的cmc-link选择器定位币种名称
    crypto_link = crypto_details.find("a", class_="cmc-link")
    if crypto_link:
        crypto_name = crypto_link.find("p")
        if crypto_name:
            print(crypto_name.text.strip())
    sleep(randint(2, 10))

driver.quit()

内容的提问来源于stack exchange，提问作者TTiM3R