Python爬取Finviz数据失效，报ValueError: No tables found求助

问题背景 #

9个月前可正常运行的Python网页爬取代码，如今触发ValueError: No tables found错误。代码原本目标是从Finviz.com的表格爬取约50列数据，并从Finviz个股页面的另一表格获取Sales和Income两列插入结果。

报错原因 #

核心原因是Finviz网站页面结构或反爬机制更新：

• 原目标数据表格的class名称从snapshot-table2变更为snapshot-table，导致pd.read_html无法匹配到对应表格
• 旧版User-Agent可能被反爬机制识别，请求返回非正常页面内容

解决方案 #

1. 调整表格匹配规则 #

更新pd.read_html的匹配参数，适配当前Finviz页面的表格class名称。

2. 增强反爬规避措施 #

• 使用更现代的浏览器User-Agent
• 添加请求重试机制，应对临时请求失败
• 延长随机休眠时间，降低被封禁风险

修改后的完整代码 #

import pandas as pd
import requests
import bs4
import time
import random
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

# 替换为现代浏览器标识的User-Agent
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
}

# 配置请求重试，应对临时服务器错误或限流
session = requests.Session()
retry = Retry(total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504])
adapter = HTTPAdapter(max_retries=retry)
session.mount('https://', adapter)
session.mount('http://', adapter)

def testDf(version):
    url = 'https://finviz.com/screener.ashx?v={version}&r={page}&f=all&c=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70&f=ind_stocksonly&o=-marketcap'
    
    page = 1

    screen = session.get(url.format(version=version, page=page), headers=headers)
    soup = bs4.BeautifulSoup(screen.text, features='lxml')
    pages = int(soup.find_all('a', {'class': 'screener-pages'})[-1].text)

    data = []
    for page in range(1, 20 * pages, 20):
        print(version, page)
        screen = session.get(url.format(version=version, page=page), headers=headers).text
        tables = pd.read_html(screen)
        tables = tables[-2]
        tables.columns = tables.iloc[0]
        tables = tables[1:]
        data.append(tables)
        # 延长随机休眠时间，降低反爬触发概率
        time.sleep(random.uniform(1, 3))
    return pd.concat(data).reset_index(drop=True).rename_axis(columns=None)


df = testDf('152').copy()
df.insert(2,'Sales','')
df.insert(3,'Income','')

def get_ticker(ticker):
    url = f'https://finviz.com/quote.ashx?t={ticker.upper()}'
    
    req = session.get(url, headers=headers)
    # 更新表格class为当前网站使用的snapshot-table
    table = pd.read_html(req.text, attrs={"class": "snapshot-table"})
    df_table = table[0]
    # 确认数据位置，当前Sales对应索引3，Income对应索引2
    return (df_table[1][3], df_table[1][2])


df['Sales'],df['Income'] = zip(*df['Ticker'].map(lambda x: get_ticker(x)))

print(df)

# 导出结果到Excel
writer = pd.ExcelWriter("Gold.xlsx", engine='xlsxwriter')
df.to_excel(writer, sheet_name='Sheet1', index=False)

workbook  = writer.book
worksheet = writer.sheets['Sheet1']

header_format = workbook.add_format()
header_format.set_font_name('Calibri')
header_format.set_font_color('green')
header_format.set_font_size(8)
header_format.set_italic()
header_format.set_underline()

for col_num, value in enumerate(df.columns.values):
    worksheet.write(0, col_num, value, header_format)

writer.close()

额外注意事项 #

• 若再次出现表格未找到的错误，需手动访问Finviz个股页面，检查表格class名称是否再次变更
• 如果遇到验证码页面，说明IP被反爬机制识别，需更换IP或添加代理配置

内容的提问来源于stack exchange，提问作者user3444610