Python爬取Finviz数据异常：数值变为NaN求排查

问题背景 #

今年2月运行爬虫代码时，可正常从Finviz获取股票Sales、Income数据：

Sales     Income
AAPL   365.82B     94.68B
MSFT   184.90B     71.19B
TSLA    53.82B      5.52B
FB     112.33B     40.30B

现在运行代码后，仅AAPL数据正常，其余股票对应字段均为NaN：

Sales     Income
AAPL   365.82B     94.68B
MSFT   NaN          NaN
TSLA   NaN          NaN
FB     NaN          NaN

网站表格结构无变化，以下是原代码：

import pandas as pd
from bs4 import BeautifulSoup as bs
import requests
import numpy as np

# For custom list of stocks, edit this list below, otherwise leave commented out
v1 = ['AAPL','MSFT','TSLA','FB','BRK-B','TSM','NVDA','V','JNJ','JPM','WMT','PG','BAC','HD','BABA','TM','XOM','PFE','DIS','KO']
# Header required to scrape from Finviz
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36',
           'Upgrade-Insecure-Requests': '1', 'Cookie': 'v2=1495343816.182.19.234.142', 'Accept-Encoding': 'gzip, deflate, sdch',
           'Referer': "http://finviz.com/quote.ashx?t="}
    
# This function is what is used to find the metric of interest and return it
def fundamental_metric(soup, metric):
    return soup.find(text=metric).find_next(class_='snapshot-td2').text
    
# This function iterates through the index of the data frame (stock_list) and uses the fundemental_metric functinon to find the metric on Finviz for that stock
# Any stock in the list that cannot be scraped will return an error before moving on to the next stock
def get_fundamental_data(df):
    for symbol in df.index:
        try:
            #url = ("http://finviz.com/quote.ashx?t=" + symbol.lower())
            r = requests.get("http://finviz.com/quote.ashx?t="+ symbol.lower(),headers=headers)
            soup = bs(r.content,'html.parser')
            for m in df.columns:
                output = fundamental_metric(soup,m)
                df.loc[symbol,m] = output
                df.replace(['-'], np.NaN)
        except Exception as e:
            print (symbol, 'Not Found')
            print(e)
        return df
    
# List of metrics to scrape
# Before adding any metrics, ensure the metric being added is available on Finviz and the name is matched identically
metric = ['Sales','Income']
    
df = pd.DataFrame(index = v1, columns = metric)
df = get_fundamental_data(df)
    
print(df)

问题排查与修复 #

1. 核心bug：循环提前返回，仅处理第一个股票 #

get_fundamental_data函数中，return df被放在了for循环内部（try/except块之后）。这导致代码处理完第一个股票（AAPL）就直接返回DataFrame，后续股票完全没被执行，自然保持初始的NaN值。

2. 反爬拦截：Cookie过期 #

代码中使用的Cookie是固定旧值v2=1495343816.182.19.234.142，这类会话Cookie早已过期。Finviz的反爬机制会拦截无有效会话的请求，返回不完整页面，导致soup.find(text=metric)找不到目标元素，触发异常后数据留空。

3. 次要问题：replace操作未生效 #

df.replace(['-'], np.NaN)没有设置inplace=True，也未重新赋值给df，所以无法将页面中的-转换为NaN。

修正后的代码 #

import pandas as pd
from bs4 import BeautifulSoup as bs
import requests
import numpy as np
import time

# 自定义股票列表
v1 = ['AAPL','MSFT','TSLA','FB','BRK-B','TSM','NVDA','V','JNJ','JPM','WMT','PG','BAC','HD','BABA','TM','XOM','PFE','DIS','KO']

# 更新请求头：移除过期Cookie，使用主流浏览器UA
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36',
    'Upgrade-Insecure-Requests': '1',
    'Accept-Encoding': 'gzip, deflate, br',
    'Referer': "http://finviz.com/quote.ashx?t="
}

# 获取指定指标的函数，增加异常捕获
def fundamental_metric(soup, metric):
    try:
        return soup.find(text=metric).find_next(class_='snapshot-td2').text
    except AttributeError:
        return np.NaN

# 批量获取股票数据：循环结束后再返回
def get_fundamental_data(df):
    for symbol in df.index:
        try:
            url = f"http://finviz.com/quote.ashx?t={symbol.lower()}"
            r = requests.get(url, headers=headers)
            r.raise_for_status()  # 检查请求是否成功
            soup = bs(r.content, 'html.parser')
            
            for m in df.columns:
                output = fundamental_metric(soup, m)
                df.loc[symbol, m] = output
            
            # 执行替换操作并生效
            df.replace(['-'], np.NaN, inplace=True)
            # 添加延迟避免反爬
            time.sleep(1)
        except Exception as e:
            print(f"{symbol} 处理失败: {str(e)}")
    # 所有股票处理完再返回
    return df

# 要抓取的指标列表
metrics = ['Sales','Income']

df = pd.DataFrame(index=v1, columns=metrics)
df = get_fundamental_data(df)

print(df)

内容的提问来源于stack exchange，提问作者vinny russo