使用BeautifulSoup解析网页无法稳定获取最新更新，是代码问题吗？

我写了一段代码用来监测指定网页的实时更新（页面有定期发布更新的板块），但代码没法稳定返回最新内容：首次运行总能拿到最新内容，但循环运行时会随机返回第二或第三新的更新。我觉得不是网站本身的问题，网页格式一直没变，就是拿不到'Live Updates:'文本后的最新<p>标签。

我的代码 #

import requests
from bs4 import BeautifulSoup
from groupy.client import Client
import time

# 要监测更新的网页URL
url = "https://www.israelnationalnews.com/news/378017"
updates_list = []

# 下载网页内容的函数
def download_webpage(url):
    try:
        response = requests.get(url)
        response.raise_for_status()  # 遇到HTTP错误抛出异常
        return response.text
    except requests.exceptions.RequestException as e:
        print("Error:", e)
        return None

# 检查'<strong>Live Updates:</strong>'下的更新
def check_for_alerts(html_content):
    if html_content:
        soup = BeautifulSoup(html_content, 'html.parser')
        live_updates_tag = soup.find('p', class_='', text='Live Updates:')
        
        if live_updates_tag:
            next_p_element = live_updates_tag.find_next('p')
            if next_p_element:
                print(f"\nUpdate found:\n{next_p_element.get_text(strip=True)}")
            else:
                print("No updates found under 'Live Updates:'")
        else:
            print("No 'Live Updates:' found on the webpage.")
    else:
        print("Webpage content not found.")
    
    update = next_p_element.get_text(strip=True)
    return update

def check_recent(update):
    if update in updates_list:
        print ("This latest update is not new.\n")
    elif update not in updates_list:
        updates_list.append(update)

        client = Client.from_token('I3TymkKoSXxNrg1vCpouYZb4HVH2LO3X54NPPptC')
        chat = client.groups.get('97129195')
        message = chat.post(text=f"Arutz Sheva update: {update}")

if __name__ == "__main__":
    while True:
        webpage_content = download_webpage(url)
        update = check_for_alerts(webpage_content)
        check_recent(update)

        time.sleep(10)

示例输出 #

Update found:
Monday: 8:50 p.m.Dozens of IAF aircraft have attacked in the last few hours and continue to attack many targets throughout the Rimal neighborhood in Gaza.

Update found:
Monday: 8:50 p.m.Dozens of IAF aircraft have attacked in the last few hours and continue to attack many targets throughout the Rimal neighborhood in Gaza.
This latest update is not new.


Update found:
Monday: 8:00 p.m.The official death toll from the Hamas massacre was raised to at least 900.
This latest update is not new.


Update found:
Monday: 8:50 p.m.Dozens of IAF aircraft have attacked in the last few hours and continue to attack many targets throughout the Rimal neighborhood in Gaza.
This latest update is not new.


Update found:
Monday: 8:50 p.m.Dozens of IAF aircraft have attacked in the last few hours and continue to attack many targets throughout the Rimal neighborhood in Gaza.
This latest update is not new.


Update found:
Monday: 8:50 p.m.Dozens of IAF aircraft have attacked in the last few hours and continue to attack many targets throughout the Rimal neighborhood in Gaza.
This latest update is not new.


Update found:
Monday: 8:50 p.m.Dozens of IAF aircraft have attacked in the last few hours and continue to attack many targets throughout the Rimal neighborhood in Gaza.
This latest update is not new.


Update found:
Monday: 8:50 p.m.Dozens of IAF aircraft have attacked in the last few hours and continue to attack many targets throughout the Rimal neighborhood in Gaza.
This latest update is not new.


Update found:
Monday: 8:50 p.m.Dozens of IAF aircraft have attacked in the last few hours and continue to attack many targets throughout the Rimal neighborhood in Gaza.
This latest update is not new.


Update found:
Monday: 8:50 p.m.Dozens of IAF aircraft have attacked in the last few hours and continue to attack many targets throughout the Rimal neighborhood in Gaza.
This latest update is not new.

我试过多种HTML解析方式，但问题依旧。

解决方案 #

问题根源分析 #

1. 请求缓存问题：requests.get()默认会利用缓存机制，循环请求时可能拿到旧页面内容，导致获取不到最新更新。
2. 元素定位不稳定：用text='Live Updates:'精确匹配文本，可能因为文本嵌套在<strong>标签内或存在空格/换行符，导致匹配失败；find_next('p')仅取第一个后续<p>，若拿到旧页面就会返回旧更新。
3. 变量作用域隐患：如果live_updates_tag不存在，next_p_element未定义，直接调用get_text()会抛出异常。
4. 低效的API初始化：每次检测新更新都重新创建GroupMe客户端，增加不必要开销。

修改后的代码 #

import requests
from bs4 import BeautifulSoup
from groupy.client import Client
import time

# 要监测更新的网页URL
url = "https://www.israelnationalnews.com/news/378017"
updates_list = []
# 提前初始化GroupMe客户端，避免重复创建
client = Client.from_token('I3TymkKoSXxNrg1vCpouYZb4HVH2LO3X54NPPptC')
chat = client.groups.get('97129195')

# 下载网页内容的函数，禁用缓存
def download_webpage(url):
    headers = {
        'Cache-Control': 'no-cache',
        'Pragma': 'no-cache',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
    }
    try:
        response = requests.get(url, headers=headers)
        response.raise_for_status()  # 遇到HTTP错误抛出异常
        return response.text
    except requests.exceptions.RequestException as e:
        print("Error:", e)
        return None

# 检查'<strong>Live Updates:</strong>'下的更新，优化元素定位
def check_for_alerts(html_content):
    if not html_content:
        print("Webpage content not found.")
        return None
    
    soup = BeautifulSoup(html_content, 'html.parser')
    # 找包含Live Updates:文本的p标签（兼容嵌套标签和空格）
    live_updates_tag = soup.find('p', class_='', string=lambda text: text and 'Live Updates:' in text.strip())
    
    if not live_updates_tag:
        print("No 'Live Updates:' found on the webpage.")
        return None
    
    # 获取Live Updates标签之后的所有p元素，取第一个（最新更新）
    next_p_elements = live_updates_tag.find_next_siblings('p')
    if not next_p_elements:
        print("No updates found under 'Live Updates:'")
        return None
    
    latest_update = next_p_elements[0].get_text(strip=True)
    print(f"\nUpdate found:\n{latest_update}")
    return latest_update

def check_recent(update):
    if not update:
        return
    
    if update in updates_list:
        print ("This latest update is not new.\n")
    else:
        updates_list.append(update)
        message = chat.post(text=f"Arutz Sheva update: {update}")
        print("New update posted to GroupMe!\n")

if __name__ == "__main__":
    # 首次运行先获取一次最新更新加入列表，避免重复推送
    initial_content = download_webpage(url)
    initial_update = check_for_alerts(initial_content)
    if initial_update:
        updates_list.append(initial_update)
    
    while True:
        webpage_content = download_webpage(url)
        update = check_for_alerts(webpage_content)
        check_recent(update)

        time.sleep(10)

修改说明 #

1. 禁用缓存：添加请求头强制禁用缓存，同时模拟浏览器UA，确保拿到最新页面内容。
2. 优化元素定位：用lambda表达式模糊匹配目标文本，兼容嵌套标签和空格；通过find_next_siblings获取所有更新段落，取第一个即为最新内容。
3. 修复异常隐患：增加多环节空值判断，避免因元素未找到抛出异常。
4. 优化API调用：提前初始化GroupMe客户端和聊天对象，减少重复初始化开销。
5. 首次运行处理：启动时先获取一次最新更新加入列表，避免首次循环重复推送。

内容的提问来源于stack exchange，提问作者Imcha Anochi