循环批量爬取URL列表时如何避免‘No Tables Found’错误？

问题背景 #

单个站点爬取指定表格的代码可正常运行，但添加URL路径列表和for循环实现批量爬取时，出现No Tables Found错误。

单个站点爬取代码 #

import io
import re
import pandas as pd
import requests
from bs4 import BeautifulSoup

url = "https://magicseaweed.com/Belmar-Surf-Report/3683/"
html = requests.get(url).content
soup = BeautifulSoup(html, "html.parser")

# table 1
regex = re.compile("^table table-primary.*")
table1 = soup.find("table", {"class": regex})
df1 = pd.read_html(io.StringIO(str(table1)))[
    0].iloc[:9, [0, 1, 2, 3, 4, 6, 7, 12, 15]]
res1 = [df1.columns.values.tolist(), *df1.values.tolist()]
# print(res1)

# table 2
tables = soup.findAll("table")
table2 = tables[0]
df2 = pd.read_html(io.StringIO(str(table2)))[0]
res2 = df2.values.tolist()
# print(res2)

# table 3
table3 = tables[1]
df3 = pd.read_html(io.StringIO(str(table3)))[0]
res3 = df3.values.tolist()
print(res3)

批量爬取报错代码 #

import io
import re
import pandas as pd
import requests
from bs4 import BeautifulSoup


id_list = [
    '/Belmar-Surf-Report/3683',
    '/Manasquan-Surf-Report/386/',
    '/Ocean-Grove-Surf-Report/7945/',
    '/Asbury-Park-Surf-Report/857/',
    '/Avon-Surf-Report/4050/',
    '/Bay-Head-Surf-Report/4951/',
    '/Belmar-Surf-Report/3683/',
    '/Boardwalk-Surf-Report/9183/',
      
]


for x in id_list:

    url = 'https://magicseaweed.com' + x
    html = requests.get(url).content
    soup = BeautifulSoup(html, "html.parser")

    # table 1
    regex = re.compile("^table table-primary.*")
    table1 = soup.find("table", {"class": regex})
    df1 = pd.read_html(io.StringIO(str(table1)))[
        0].iloc[:9, [0, 1, 2, 3, 4, 6, 7, 12, 15]]
    res1 = [df1.columns.values.tolist(), *df1.values.tolist()]
    print(res1)

    # table 2
    tables = soup.findAll("table")
    table2 = tables[0]
    df2 = pd.read_html(io.StringIO(str(table2)))[0]
    res2 = df2.values.tolist()
    print(res2)

    # table 3
    table3 = tables[1]
    df3 = pd.read_html(io.StringIO(str(table3)))[0]
    res3 = df3.values.tolist()
    print(res3)

错误原因分析 #

• URL格式不统一：id_list中部分路径末尾带斜杠，部分不带。虽然多数服务器会重定向，但重定向后的页面可能存在结构变化，或请求返回内容不符合预期，导致后续表格查找失败。
• 缺乏异常处理机制：循环中未判断请求是否成功（如遇到404、500状态码或反爬拦截），若返回页面无表格，直接调用pd.read_html会触发No Tables Found错误。
• 表格索引依赖不可靠：通过tables[0]、tables[1]固定索引取表格，不同页面的表格数量和顺序可能不一致，一旦页面表格数量不足2个，会引发索引越界，或取到错误的非目标表格。
• 正则匹配逻辑有缺陷：^table table-primary.*的正则要求class属性以table table-primary开头，但HTML中class的顺序可能变化（比如table-primary table），或存在其他前缀，导致匹配不到目标表格，table1为None，传给pd.read_html后报错。

修正后的代码 #

import io
import re
import pandas as pd
import requests
from bs4 import BeautifulSoup

# 统一URL格式，确保末尾无多余斜杠
id_list = [
    '/Belmar-Surf-Report/3683',
    '/Manasquan-Surf-Report/386',
    '/Ocean-Grove-Surf-Report/7945',
    '/Asbury-Park-Surf-Report/857',
    '/Avon-Surf-Report/4050',
    '/Bay-Head-Surf-Report/4951',
    '/Belmar-Surf-Report/3683',
    '/Boardwalk-Surf-Report/9183',
]

# 添加请求头，模拟浏览器访问，避免反爬拦截
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
}

for idx, path in enumerate(id_list):
    try:
        # 拼接URL并确保格式正确
        url = f"https://magicseaweed.com{path}/" if not path.endswith('/') else f"https://magicseaweed.com{path}"
        response = requests.get(url, headers=headers)
        # 检查请求是否成功
        response.raise_for_status()
        soup = BeautifulSoup(response.text, "html.parser")

        # 修正table1的匹配逻辑：匹配包含table-primary类的表格，不依赖顺序
        table1 = soup.find("table", class_=lambda c: c and "table-primary" in c.split())
        if not table1:
            print(f"第{idx+1}个站点：未找到table1")
            continue
        df1 = pd.read_html(io.StringIO(str(table1)))[0].iloc[:9, [0,1,2,3,4,6,7,12,15]]
        res1 = [df1.columns.values.tolist(), *df1.values.tolist()]
        print(f"第{idx+1}个站点table1结果：", res1)

        # 获取所有表格，先检查数量是否足够
        tables = soup.find_all("table")
        if len(tables) < 2:
            print(f"第{idx+1}个站点：表格数量不足")
            continue
        
        # table2和table3处理
        df2 = pd.read_html(io.StringIO(str(tables[0])))[0]
        res2 = df2.values.tolist()
        print(f"第{idx+1}个站点table2结果：", res2)

        df3 = pd.read_html(io.StringIO(str(tables[1])))[0]
        res3 = df3.values.tolist()
        print(f"第{idx+1}个站点table3结果：", res3)

    except Exception as e:
        print(f"第{idx+1}个站点处理失败：", str(e))
    print("-"*50)

关键优化点 #

• 统一URL格式，避免重定向带来的不确定性；
• 添加请求头模拟浏览器，降低被反爬拦截的概率；
• 增加异常捕获，处理请求失败、表格缺失等情况，避免程序崩溃；
• 修正表格匹配逻辑，通过类包含判断代替严格的正则前缀匹配，提升兼容性；
• 先检查表格数量，再进行索引访问，避免索引越界错误。

内容的提问来源于stack exchange，提问作者Anthony Madle