Python用bs4爬取Coinmarketcap循环时突发NoneType不可迭代错误如何解决

报错原因 #

1. 反爬策略触发：CoinMarketCap有明确的爬虫频率限制，你连续无间隔请求70多次后被系统识别为异常访问，返回了验证码页面、429频率超限页面或者空白响应，导致页面源码里没有你要找的目标class，soup.find()返回None，遍历None就抛出了这个类型错误。你手动访问页面时是正常的浏览器环境，没有触发拦截，所以能看到对应的class。
2. 代码缺乏容错处理：你没有对请求状态、元素存在性做校验，只要一次请求异常就直接中断整个任务。
3. 默认请求头特征明显：你用的requests默认UA会被网站直接识别为爬虫，更容易触发拦截。

修复方案 #

首先导入time模块，添加请求头、请求间隔、异常校验逻辑，修改后的代码参考如下：

import requests
from bs4 import BeautifulSoup
import pandas as pd
import os
import time

path =r'C:\Users\mrfau\Documents\python\stock_screener_001\crypto_project'
urls= pd.read_csv(os.path.join(path,r'all_coin_link.csv'))

# 自定义请求头，模拟浏览器访问
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36'
}

ranks = []
names = []
prices = []
count = 0
# 记录爬取失败的链接方便后续补爬
failed_urls = []

for url in urls['coin_link']:
    count += 1
    print('Loop', count, f'started for {url}')
    rank = []
    name = []
    price = []
    try:
        # 加超时防止卡死
        r = requests.get(url, headers=headers, timeout=10)
        # 校验响应状态
        if r.status_code != 200:
            print(f'请求异常，状态码：{r.status_code}，跳过该链接')
            failed_urls.append(url)
            # 异常后多等几秒再继续
            time.sleep(3)
            continue
        soup = BeautifulSoup(r.text, 'html.parser')
        
        # 先判断元素是否存在再遍历
        rank_div = soup.find('div', class_ = 'sc-16r8icm-0 bILTHz')
        if rank_div:
            for item in rank_div:
                item = item.text
                rank.append(item)
        ranks.append(rank)

        name_h2 = soup.find('h2', class_ = 'sc-1q9q90x-0 jCInrl h1')
        if name_h2:
            for ticker in name_h2:
                ticker = ticker.text
                name.append(ticker)
        names.append(name)

        price_div = soup.find('div', class_ = 'sc-16r8icm-0 kjciSH priceTitle')
        if price_div:
            for price_tag in price_div:
                price_tag = price_tag.text
                price.append(price_tag)
        prices.append(price)
        
        # 每次请求后间隔1.5秒，降低频率
        time.sleep(1.5)
    except Exception as e:
        print(f'爬取{url}失败，错误：{str(e)}')
        failed_urls.append(url)
        time.sleep(3)
        continue

df = pd.DataFrame(ranks)
df2 = pd.DataFrame(names)
df3 = pd.DataFrame(prices)
final_table = pd.concat([df, df2, df3], axis=1)
final_table.columns=['rank', 'type', 'watchlist', 'name', 'symbol', 'price', 'changes']

final_table.to_csv(os.path.join(path,r'summary.csv'))
# 保存失败的链接
pd.DataFrame(failed_urls, columns=['failed_url']).to_csv(os.path.join(path, 'failed_urls.csv'), index=False)

如果调整间隔后还是频繁被拦截，可以考虑用代理IP池进一步降低被封的概率。

内容的提问来源于stack exchange，提问作者Fauzi Yahaya