HTML表格提取问题：BeautifulSoup选择器仅单日生效求助

问题说明 #

• 使用BeautifulSoup爬取Little River Outfitters网站钓鱼数据时，2018年8月2日的数据能正常提取表格内容，但8月3日至8日的数据无法获取
• 原CSS选择器为table table tr:has(img[alt*="Fishing Gauge"]) td:has(p)，尝试替换为blockquote p:has(img[alt*="Fishing Gauge"])后仍无效，需适配后续日期的页面结构以获取完整数据

原代码 #

from bs4 import BeautifulSoup
import pandas as pd
from datetime import date, timedelta
import requests
import os

start_date = date(2018,8,2)
end_date = date(2018,8,8)

data = []
while True:
    start_date.strftime("%m%d%y")

    url = f'https://littleriveroutfitters.com/WEBSITE-2008/pages/fishing/{start_date.strftime("%m%d%y")}.htm'
    
    try:
        soup = BeautifulSoup(requests.get(url).text)

        d = soup.select('table table tr:has(img[alt*="Fishing Gauge"]) td:has(p)')
        d = dict(zip(d[0].stripped_strings, d[1].stripped_strings))
        d.update({'date': start_date})
        data.append(d)
    except:
        print(start_date, url)

    if start_date == end_date:
        break
    else:
        start_date = start_date + timedelta(days=1)


df_1 = pd.DataFrame(data)

save_path = "C:/Users/Bay/Documents/Python_Files/fish_info_scrape/"

#df_1.to_csv(save_path + 'report_output.csv', mode = 'w')

df_1

当前输出 #

Water Temperature Little River        Stream Flow Sunrise Sunset Rainfall 2018 YTD Knoxville Apt Rainfall Normal YTD Knoxville Apt        date       
0                64.4 Fahrenheit  2.93 Feet 678 CFS    6:44   8:39                          33.62"                            30.45"  2018-08-02       
1                            NaN                NaN     NaN    NaN                             NaN                               NaN  2018-08-03       
2                            NaN                NaN     NaN    NaN                             NaN                               NaN  2018-08-04       
3                            NaN                NaN     NaN    NaN                             NaN                               NaN  2018-08-05       
4                            NaN                NaN     NaN    NaN                             NaN                               NaN  2018-08-06       
5                            NaN                NaN     NaN    NaN                             NaN                               NaN  2018-08-07       
6                            NaN                NaN     NaN    NaN                             NaN                               NaN  2018-08-08       

解决方案 #

问题核心是8月2日后的页面结构发生了变化，原选择器无法匹配新结构。以下是修改后的代码，通过多选择器降级和灵活的数据提取逻辑适配不同页面：

from bs4 import BeautifulSoup
import pandas as pd
from datetime import date, timedelta
import requests
import os

start_date = date(2018,8,2)
end_date = date(2018,8,8)

data = []
while True:
    date_str = start_date.strftime("%m%d%y")
    url = f'https://littleriveroutfitters.com/WEBSITE-2008/pages/fishing/{date_str}.htm'
    
    try:
        response = requests.get(url)
        response.raise_for_status()  # 捕获HTTP请求错误
        soup = BeautifulSoup(response.text, 'html.parser')
        
        # 优先尝试原选择器
        d_elements = soup.select('table table tr:has(img[alt*="Fishing Gauge"]) td:has(p)')
        if not d_elements:
            # 备选方案：定位Fishing Gauge图片，提取其父容器内的文本数据
            gauge_img = soup.find('img', alt=lambda x: x and 'Fishing Gauge' in x)
            if gauge_img:
                parent_block = gauge_img.find_parent('blockquote')
                if parent_block:
                    # 提取所有有效文本，按键值对交替分组
                    text_items = [item.strip() for item in parent_block.stripped_strings if item.strip()]
                    d_elements = [text_items[::2], text_items[1::2]]
        
        # 组装数据
        if d_elements and len(d_elements) >=2:
            d = dict(zip(d_elements[0], d_elements[1]))
            d.update({'date': start_date})
            data.append(d)
        else:
            data.append({'date': start_date})
            print(f"无有效数据: {start_date} {url}")
    
    except Exception as e:
        print(f"请求/解析失败: {start_date} {url} - {str(e)}")

    if start_date == end_date:
        break
    else:
        start_date = start_date + timedelta(days=1)


df_1 = pd.DataFrame(data)

save_path = "C:/Users/Bay/Documents/Python_Files/fish_info_scrape/"
# df_1.to_csv(save_path + 'report_output.csv', mode='w')

print(df_1)

修改说明 #

1. 增加请求校验：用response.raise_for_status()排查HTTP层面的访问错误
2. 多结构适配：原选择器失效时，通过定位目标图片的父容器提取文本数据
3. 灵活键值对提取：将页面文本按交替顺序自动分组为键和值，适配不同排版
4. 明确日志输出：区分无数据和请求错误，方便排查问题

内容的提问来源于stack exchange，提问作者slippysnorlax