如何按年份保留Web Archive归档URL的最新爬取版本并去重？

问题背景 #

原始Web Archive归档URL列表：

raw_urls = [
    "<GET https://web.archive.org/web/20110518083020id_/http://www.rollsroyce.com/about/index.jsp>",
    "<GET https://web.archive.org/web/20110617060411id_/http://www.rollsroyce.com/about/index.jsp>",
    "<GET https://web.archive.org/web/20110810011112id_/http://www.rollsroyce.com/about/index.jsp>",
    "<GET https://web.archive.org/web/20120120013630id_/http://rollsroyce.com/about/index.jsp>",
    "<GET https://web.archive.org/web/20120124115631id_/http://rollsroyce.com/about/index.jsp>",
    "<GET https://web.archive.org/web/20120127145524id_/http://rollsroyce.com/about/index.jsp>",
    "<GET https://web.archive.org/web/20120131060020id_/http://rollsroyce.com/about/index.jsp>",
    "<GET https://web.archive.org/web/20120210041718id_/http://rollsroyce.com/about/index.jsp>",
    "<GET https://web.archive.org/web/20120214181911id_/http://rollsroyce.com/about/index.jsp>",
    "<GET https://web.archive.org/web/20120224215742id_/http://rollsroyce.com/about/index.jsp>",
    "<GET https://web.archive.org/web/20120318055019id_/http://rollsroyce.com/about/index.jsp>",
]

需求：

• 基于id_/后的目标URL去重
• 按时间戳的年份分组，每个分组保留该年份下目标URL的最新归档版本
• 原列表不一定有序，需快速实现

快速实现方法（Python） #

核心逻辑和你设想的一致，用字典分组存储，直接比较时间戳字符串（14位数字串可直接按字典序比较大小），无需转换datetime，效率更高：

def filter_latest_archive_by_year(raw_urls):
    # 用字典存储分组：键为 (目标URL, 年份)，值为对应最新的归档URL
    groups = {}
    
    for url_str in raw_urls:
        # 清理字符串，去掉<GET 和>
        cleaned = url_str.strip("<GET >")
        # 拆分时间戳和目标URL
        timestamp_part, target_url = cleaned.split("id_/", 1)
        # 提取时间戳（web.archive.org/web/后的14位数字）
        timestamp = timestamp_part.split("/")[-1]
        # 提取年份（前4位）
        year = timestamp[:4]
        
        # 构造分组键
        key = (target_url, year)
        # 如果键不存在，或当前时间戳更大，更新为当前URL
        if key not in groups or timestamp > groups[key]["timestamp"]:
            groups[key] = {
                "timestamp": timestamp,
                "full_url": cleaned
            }
    
    # 提取结果，还原成原始格式（带<GET >）
    result = [f"<GET {item['full_url']}>" for item in groups.values()]
    return result

# 测试
filtered = filter_latest_archive_by_year(raw_urls)
print(filtered)

输出结果 #

[
    "<GET https://web.archive.org/web/20110810011112id_/http://www.rollsroyce.com/about/index.jsp>",
    "<GET https://web.archive.org/web/20120318055019id_/http://rollsroyce.com/about/index.jsp>"
]

优化点说明 #

• 直接用时间戳字符串比较大小，避免datetime转换的开销，处理大量URL时效率更高
• 字典分组自动去重，一次遍历即可完成筛选，时间复杂度O(n)
• 兼容无序的原始列表

内容的提问来源于stack exchange，提问作者Jack