如何无需点击加载更多按钮爬取Inshorts网站含指定关键词的全部新闻

实现思路 #

• inshorts的「加载更多」功能通过后端AJAX接口实现，无需模拟浏览器点击操作，直接构造合法POST请求即可批量获取历史新闻数据
• 初始页面加载时会生成news_offset隐藏字段，作为翻页请求的唯一凭证，每次请求接口会返回下一页对应的offset值
• 解析每页新闻的标题、内容后增加关键词匹配逻辑，仅留存包含目标关键词的新闻即可

完整可运行代码 #

import requests
import time
from bs4 import BeautifulSoup
import pandas as pd

# 配置参数
TARGET_KEYWORD = "COVID-19"  # 替换为你需要的关键词
CATEGORY = "national"
BASE_URL = f"https://inshorts.com/en/read/{CATEGORY}"
MORE_NEWS_API = "https://inshorts.com/en/ajax/more_news"
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    "X-Requested-With": "XMLHttpRequest"
}
# 连续N页无匹配关键词则终止爬取
STOP_THRESHOLD = 3

news_data_content, news_data_title, news_data_category, news_data_time = [], [], [], []
no_match_count = 0

# 先爬取初始页面
resp = requests.get(BASE_URL, headers=HEADERS)
soup = BeautifulSoup(resp.content, 'html.parser')

# 提取初始news_offset
news_offset = soup.find("script", id="ajax_params").get("data-news-offset")

def parse_news(soup_obj, category):
    page_match_count = 0
    for headline, article, time_block in zip(
        soup_obj.find_all('div', class_=["news-card-title news-right-box"]),
        soup_obj.find_all('div', class_=["news-card-content news-right-box"]),
        soup_obj.find_all('div', class_=["news-card-author-time news-card-author-time-in-title"])
    ):
        title = headline.find('span', attrs={'itemprop': "headline"}).string.strip()
        content = article.find('div', attrs={'itemprop': "articleBody"}).string.strip()
        pub_time = time_block.find('span', class_=["date"]).text.strip()
        
        # 关键词匹配，标题或内容包含关键词即留存
        if TARGET_KEYWORD.lower() in title.lower() or TARGET_KEYWORD.lower() in content.lower():
            news_data_title.append(title)
            news_data_content.append(content)
            news_data_time.append(pub_time)
            news_data_category.append(category)
            page_match_count += 1
    return page_match_count

# 解析初始页新闻
parse_news(soup, CATEGORY)

# 循环爬取更多页
while no_match_count < STOP_THRESHOLD and news_offset:
    # 构造POST请求参数
    payload = {
        "category": CATEGORY,
        "news_offset": news_offset
    }
    resp = requests.post(MORE_NEWS_API, headers=HEADERS, data=payload)
    resp_data = resp.json()
    
    # 解析返回的新闻HTML
    news_soup = BeautifulSoup(resp_data["html"], 'html.parser')
    match_count = parse_news(news_soup, CATEGORY)
    
    # 更新offset和无匹配计数
    news_offset = resp_data["offset"]
    if match_count == 0:
        no_match_count += 1
    else:
        no_match_count = 0
    
    # 加延时避免被封
    time.sleep(1)

# 生成CSV
df = pd.DataFrame({
    "Title": news_data_title,
    "Content": news_data_content,
    "Category": news_data_category,
    "Time": news_data_time
})

file_name = input("File Name: ")
df.to_csv(f"{file_name}.csv", index=False, encoding="utf-8-sig")

注意事项 #

• 可按需修改代码开头的TARGET_KEYWORD变量值切换你要搜索的关键词，也可修改CATEGORY切换新闻分类
• 代码默认连续3页没有匹配到目标关键词就自动终止爬取，可调整STOP_THRESHOLD变量修改该阈值
• 建议不要删除请求间隔逻辑，过高的请求频率可能会被站点临时封禁IP

内容的提问来源于stack exchange，提问作者Nafis Ahmad