BeautifulSoup关键词搜索爬取Inshorts新闻触发AttributeError如何解决

• 执行soup=soup.findAll("div",{"class":"news-card z-depth-1"})后，soup已经从BeautifulSoup对象变为findAll返回的ResultSet结果集（本质是列表），列表不存在find_all方法，直接调用就会触发AttributeError
• 代码使用了re.compile但未导入re模块，后续运行还会触发NameError
• 原有筛选逻辑错误：直接对新闻卡片列表做文本查找会丢失DOM节点结构，后续无法提取标题、正文等字段

import requests
import re
from bs4 import BeautifulSoup
import pandas as pd
from tqdm import tqdm

# 可配置参数
TARGET_URL = "https://inshorts.com/en/read/national"
KEYWORD = "health"  # 要筛选的关键词
PAGE_NUM = 10  # 要爬取的额外页数
SAVE_PATH = "inshorts_news.csv"

d = {'headlines':[],'news':[], 'date':[]}

# 爬取第一页
r = requests.get(TARGET_URL)
soup = BeautifulSoup(r.content, 'html.parser')
# 提取下一页偏移量
min_news_id = soup.findAll("script",{"type":"text/javascript"})[2].text
min_news_id = min_news_id[25:35]
# 获取所有新闻卡片
news_cards = soup.findAll("div",{"class":"news-card z-depth-1"})

# 筛选包含关键词的新闻
for card in news_cards:
    title = card.find(itemprop="headline").getText(strip=True)
    content = card.find(itemprop="articleBody").getText(strip=True)
    date = card.find(itemprop="date").getText(strip=True)
    # 关键词大小写不敏感匹配，标题或正文包含即可
    if re.search(KEYWORD, title + content, flags=re.IGNORECASE):
        d['headlines'].append(title)
        d['news'].append(content)
        d['date'].append(date)

# 爬取更多分页
for i in tqdm(range(PAGE_NUM)):
    try:
        params = {'news_offset': min_news_id}
        req = requests.post("https://inshorts.com/en/ajax/more_news",data=params)
        json_data = req.json()
        min_news_id = json_data['min_news_id']
        soup = BeautifulSoup(json_data['html'], 'html.parser')
        news_cards = soup.findAll("div",{"class":"news-card z-depth-1"})
        # 分页内容同样做关键词筛选
        for card in news_cards:
            title = card.find(itemprop="headline").getText(strip=True)
            content = card.find(itemprop="articleBody").getText(strip=True)
            date = card.find(itemprop="date").getText(strip=True)
            if re.search(KEYWORD, title + content, flags=re.IGNORECASE):
                d['headlines'].append(title)
                d['news'].append(content)
                d['date'].append(date)
    except Exception as e:
        print(f"第{i}页爬取出错：{e}")
        pass

# 保存结果
df = pd.DataFrame(d)
df.to_csv(SAVE_PATH, index=False)
print(f"共爬取到符合条件的新闻{len(df)}条，已保存到{SAVE_PATH}")

• 新增导入re模块，支持正则匹配功能
• 把爬取参数提取为公共变量，方便后续调整关键词、爬取页数等配置
• 调整筛选逻辑：遍历每个新闻卡片，先提取所有字段，再对标题+正文做大小写不敏感的关键词匹配，匹配成功才存入结果
• 分页爬取部分也增加了相同的关键词筛选逻辑，保证所有返回的新闻都符合要求
• 优化了异常捕获逻辑，出错时会打印错误信息方便排查，同时避免程序直接中断

内容的提问来源于stack exchange，提问作者Nafis Ahmad