如何批量获取AVHerald网站的事件列表分页URL

1. 分页机制说明 #

AVHerald的事件列表没有固定页码，分页依赖页面底部「Older entries」链接里的唯一offset参数（如offset=20240227190020%2B5157b896），无法通过构造URL直接生成所有分页，必须逐页抓取下一页链接。

2. Shell脚本循环抓取方案 #

以下脚本会自动从首页开始，递归抓取所有分页URL并保存到文件：

#!/bin/bash

BASE_URL="https://avherald.com"
current_url="/h?list=&opt=0"
output_file="avherald_pages.txt"

# 初始化输出文件
> "$output_file"

while true; do
    full_url="${BASE_URL}${current_url}"
    # 检查当前URL是否已抓取过
    if ! grep -q "$full_url" "$output_file"; then
        echo "$full_url" >> "$output_file"
        echo "已抓取: $full_url"
    fi

    # 提取下一页链接
    next_link=$(curl -s "$full_url" | grep -o 'href="/h?list=&opt=0&offset=[^"]*"' | sed 's/href="//;s/"//')
    
    # 无下一页则退出循环
    [ -z "$next_link" ] && break
    
    current_url="$next_link"
    # 延迟2秒避免触发反爬
    sleep 2
done

echo "所有分页URL已保存到$output_file"

3. Python稳定抓取方案 #

如果HTML结构复杂，用Python结合BeautifulSoup更可靠：

import requests
from bs4 import BeautifulSoup
import time

BASE_URL = "https://avherald.com"
current_path = "/h?list=&opt=0"
page_urls = set()
output_file = "avherald_pages.txt"

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}

while True:
    full_url = f"{BASE_URL}{current_path}"
    if full_url in page_urls:
        break
    
    page_urls.add(full_url)
    print(f"已抓取: {full_url}")
    
    try:
        resp = requests.get(full_url, headers=headers)
        resp.raise_for_status()
        soup = BeautifulSoup(resp.text, "html.parser")
        
        # 定位「Older entries」链接
        next_btn = soup.find("a", string="Older entries")
        if not next_btn:
            break
        
        current_path = next_btn.get("href")
        time.sleep(2)
    except Exception as e:
        print(f"请求失败: {str(e)}")
        break

# 保存结果到文件
with open(output_file, "w") as f:
    for url in sorted(page_urls):
        f.write(f"{url}\n")

print(f"共抓取{len(page_urls)}条分页URL，已保存到{output_file}")

注意事项 #

• 必须保留sleep延迟，否则可能被网站封禁IP
• 若后续网站页面结构更新，需调整链接定位的规则

内容的提问来源于stack exchange，提问作者Sarah