如何用Beautiful Soup或改进哈希脚本监控网页指定区域变化？

问题描述 #

我需要监控若干URL的内容变化（比如新增文章），让脚本持续运行，内容变化时发出提示，最好能显示具体变化内容。我原本写了一段通过请求URL生成哈希值、间隔对比的脚本，但页面上层内容变化时会大量误报，我只想监控特定子区域的变化。

请问：能否用BeautifulSoup实现这类监控，知晓何时新增文章？或者有没有办法修改现有脚本，不依赖全页哈希，只监控指定子区域？

原脚本如下：

# Importing libraries
import time
import hashlib
import logging
from urllib.request import urlopen, Request

logging.basicConfig(
    format='%(asctime)s %(levelname)-8s %(message)s',
    level=logging.INFO,
    datefmt='%Y-%m-%d %H:%M:%S')

# setting the URL you want to monitor
url = Request('Input URL here',
            headers={'User-Agent': 'Mozilla/5.0'})

# to perform a GET request and load the
# content of the website and store it in a var
response = urlopen(url).read()

# to create the initial hash
currentHash = hashlib.sha224(response).hexdigest()
print("running")
time.sleep(10)
while True:
    try:
        # perform the get request and store it in a var
        response = urlopen(url).read()

        # create a hash
        currentHash = hashlib.sha224(response).hexdigest()

        # wait for 30 seconds
        time.sleep(30)

        # perform the get request
        response = urlopen(url).read()

        # create a new hash
        newHash = hashlib.sha224(response).hexdigest()

        # check if new hash is same as the previous hash
        if newHash == currentHash:
            continue

        # if something changed in the hashes
        else:
            # notify
            logging.info("something changed in given URL")

            # again read the website
            response = urlopen(url).read()

            # create a hash
            currentHash = hashlib.sha224(response).hexdigest()

            # wait for 30 seconds
            time.sleep(30)
            continue

    # To handle exceptions
    except Exception as e:
        logging.info("error")

方案一：用BeautifulSoup提取指定区域生成哈希对比 #

核心思路是放弃全页哈希，用BeautifulSoup定位到目标子区域（比如文章列表容器），仅对该区域的内容生成哈希，彻底避免页面其他部分变化导致的误报。

修改后的脚本：

import time
import hashlib
import logging
from urllib.request import urlopen, Request
from bs4 import BeautifulSoup

logging.basicConfig(
    format='%(asctime)s %(levelname)-8s %(message)s',
    level=logging.INFO,
    datefmt='%Y-%m-%d %H:%M:%S')

# 监控配置
TARGET_URL = 'Input URL here'
# 替换为目标区域的CSS选择器（可通过浏览器开发者工具获取）
MONITOR_SELECTOR = '.article-list-container'
CHECK_INTERVAL = 30  # 检查间隔（秒）

def get_target_content():
    """获取指定监控区域的内容"""
    req = Request(TARGET_URL, headers={'User-Agent': 'Mozilla/5.0'})
    response = urlopen(req).read()
    soup = BeautifulSoup(response, 'html.parser')
    target_area = soup.select_one(MONITOR_SELECTOR)
    if not target_area:
        logging.error("未找到指定监控区域")
        return None
    # 返回区域的结构化文本（或用target_area.prettify()获取HTML结构，更精准）
    return target_area.get_text(strip=True)

# 初始化监控状态
current_content = get_target_content()
if not current_content:
    exit(1)
current_hash = hashlib.sha224(current_content.encode()).hexdigest()
logging.info("监控已启动")

while True:
    try:
        time.sleep(CHECK_INTERVAL)
        new_content = get_target_content()
        if not new_content:
            continue
        
        new_hash = hashlib.sha224(new_content.encode()).hexdigest()
        if new_hash != current_hash:
            logging.info("指定区域内容发生变化！")
            # 可选：打印内容差异片段
            logging.info(f"原内容片段：{current_content[:500]}...")
            logging.info(f"新内容片段：{new_content[:500]}...")
            # 更新监控状态
            current_hash = new_hash
            current_content = new_content
        else:
            logging.debug("指定区域无变化")
    except Exception as e:
        logging.error(f"请求出错：{str(e)}")

注意事项：

1. 先安装依赖：pip install beautifulsoup4
2. 根据目标网站的HTML结构，修改MONITOR_SELECTOR为实际的区域选择器
3. 若需更精确的对比，可改用target_area.prettify()获取区域的HTML结构，避免文本格式微调导致的误报

方案二：直接监控文章列表的新增项（更直观） #

如果你的核心需求是监控新增文章，直接提取文章的唯一标识（如标题、链接），对比前后列表的差异，能直接显示新增内容，比哈希对比更精准。

示例脚本：

import time
import logging
from urllib.request import urlopen, Request
from bs4 import BeautifulSoup

logging.basicConfig(
    format='%(asctime)s %(levelname)-8s %(message)s',
    level=logging.INFO,
    datefmt='%Y-%m-%d %H:%M:%S')

# 监控配置
TARGET_URL = 'Input URL here'
# 单篇文章的容器选择器
ARTICLE_ITEM_SELECTOR = '.article-item'
# 文章标题的选择器（或用链接作为唯一标识）
ARTICLE_TITLE_SELECTOR = '.article-title'
CHECK_INTERVAL = 30

def get_article_list():
    """提取当前页面的文章标题列表"""
    req = Request(TARGET_URL, headers={'User-Agent': 'Mozilla/5.0'})
    response = urlopen(req).read()
    soup = BeautifulSoup(response, 'html.parser')
    articles = soup.select(ARTICLE_ITEM_SELECTOR)
    article_titles = []
    for item in articles:
        title_elem = item.select_one(ARTICLE_TITLE_SELECTOR)
        if title_elem:
            article_titles.append(title_elem.get_text(strip=True))
    return article_titles

# 初始化文章列表
current_articles = get_article_list()
if not current_articles:
    logging.error("未提取到文章列表")
    exit(1)
logging.info(f"初始监控到 {len(current_articles)} 篇文章")

while True:
    try:
        time.sleep(CHECK_INTERVAL)
        new_articles = get_article_list()
        if not new_articles:
            continue
        
        # 找出新增的文章
        added_articles = [title for title in new_articles if title not in current_articles]
        if added_articles:
            logging.info(f"发现新增文章！共 {len(added_articles)} 篇：")
            for title in added_articles:
                logging.info(f"- {title}")
            # 更新当前文章列表
            current_articles = new_articles
        else:
            logging.debug("无新增文章")
    except Exception as e:
        logging.error(f"请求出错：{str(e)}")

优化建议：

• 若存在标题重复的情况，可改用文章链接（title_elem['href']）作为唯一标识
• 根据网站结构调整选择器，确保能准确提取文章项

内容的提问来源于stack exchange，提问作者Ethan Combs