如何用BeautifulSoup爬取多网页活动日期并导出为CSV文件

前置准备 #

先安装需要的第三方库：

pip install requests beautifulsoup4 pandas

核心逻辑说明 #

你给出的三个站点页面结构互不相同，所以需要分别为每个站点编写对应的内容提取规则，整体流程为：

• 模拟浏览器发送请求获取页面源码
• 用BeautifulSoup解析源码，提取对应活动名称、日期字段
• 汇总所有站点的提取结果，统一导出为CSV文件

完整可运行代码 #

import time
import requests
from bs4 import BeautifulSoup
import pandas as pd

# 通用请求头，模拟浏览器避免被反爬拦截
HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
}

# 存储所有爬取到的活动
all_events = []

def crawl_monday_events(url):
    resp = requests.get(url, headers=HEADERS, timeout=10)
    soup = BeautifulSoup(resp.text, "html.parser")
    # 提取活动项，根据站点实际结构调整选择器
    event_items = soup.select(".ir_event_list_item")
    for item in event_items:
        try:
            date = item.select_one(".ir_event_list_date").get_text(strip=True)
            name = item.select_one(".ir_event_list_title").get_text(strip=True)
            all_events.append({
                "活动名称": name,
                "活动日期": date,
                "来源站点": "monday investor relations"
            })
        except Exception as e:
            # 跳过解析失败的项
            continue

def crawl_okta_events(url):
    resp = requests.get(url, headers=HEADERS, timeout=10)
    soup = BeautifulSoup(resp.text, "html.parser")
    event_items = soup.select(".events-list-item")
    for item in event_items:
        try:
            date = item.select_one(".event-date").get_text(strip=True)
            name = item.select_one(".event-title").get_text(strip=True)
            all_events.append({
                "活动名称": name,
                "活动日期": date,
                "来源站点": "okta investor relations"
            })
        except Exception as e:
            continue

def crawl_atlassian_events(url):
    resp = requests.get(url, headers=HEADERS, timeout=10)
    soup = BeautifulSoup(resp.text, "html.parser")
    event_items = soup.select(".module_item")
    for item in event_items:
        try:
            date = item.select_one(".module_date-time").get_text(strip=True)
            name = item.select_one(".module_headline").get_text(strip=True)
            all_events.append({
                "活动名称": name,
                "活动日期": date,
                "来源站点": "atlassian investor relations"
            })
        except Exception as e:
            continue

if __name__ == "__main__":
    urls = [
        "https://ir.monday.com/news-and-events/events",
        "https://investor.okta.com/events",
        "https://investors.atlassian.com/events-and-presentations/default.aspx"
    ]
    # 按站点对应调用处理函数
    crawl_monday_events(urls[0])
    time.sleep(2) # 加延时避免请求过快
    crawl_okta_events(urls[1])
    time.sleep(2)
    crawl_atlassian_events(urls[2])
    # 导出为CSV
    df = pd.DataFrame(all_events)
    df.to_csv("events.csv", index=False, encoding="utf-8-sig")
    print(f"爬取完成，共获取到{len(all_events)}条活动信息，已保存为events.csv")

常见问题说明 #

• 如果运行后获取不到内容，先检查站点是否更新了页面结构，可以右键查看网页源码，更新代码里对应的CSS选择器即可
• 如果出现403/404错误，可以尝试更新请求头里的User-Agent为你自己浏览器的UA
• 若后续站点改成动态渲染内容，再考虑替换为Selenium等动态爬取工具

内容的提问来源于stack exchange，提问作者aussieninja