如何用Python从Common Crawl WARC提取指定网页原始HTML

我已经能通过代码查询指定基准URL在Common Crawl索引中的所有条目并保存到文件，还能获取指定文章的相关信息。但已知目标数据的offset和length时，不知道怎么从对应的crawl-data（输出中的filename）里提取该文章的原始HTML，希望找到类似warcio命令的Python实现方式（用requests和warcio库）。

现有查询代码及输出 #

查询索引的代码 #

import requests
import pathlib
import json
from pprint import pprint

news_website_base = 'hobbsnews.com'
URL = "https://index.commoncrawl.org/CC-MAIN-2022-05-index?url="+news_website_base+"/*&output=json"
website_output = requests.get(URL)
pathlib.Path('data.json').write_bytes(website_output.content)

news_articles = []
test_article_num=300
for line in open('data.json', 'r'):
    news_articles.append(json.loads(line))

pprint(news_articles[test_article_num]) 

news_URL=news_articles[test_article_num]['url']
news_warc_file=news_articles[test_article_num]['filename']
news_offset=news_articles[test_article_num]['offset']
news_length=news_articles[test_article_num]['length']

代码输出 #

条目详情

{
    "digest": "GY2UDG4G3V3S5TXDL3H7HE6VCSRBD3XR",
    "filename": "crawl-data/CC-MAIN-2022-05/segments/1642320303729.69/crawldiagnostics/CC-MAIN-20220122012907-20220122042907-00614.warc.gz",
    "length": "40062",
    "mime": "text/html",
    "mime-detected": "text/html",
    "offset": "21016412",
    "status": "404",
    "timestamp": "20220122015439",
    "url": "https://www.hobbsnews.com/2020/03/22/no-new-positive-covid-19-tests-in-lea-in-last-24-hours/%7B%7B%20data.link",
    "urlkey": "com,hobbsnews)/2020/03/22/no-new-positive-covid-19-tests-in-lea-in-last-24-hours/{{%20data.link"
}

提取的关键参数

https://www.hobbsnews.com/2020/03/22/no-new-positive-covid-19-tests-in-lea-in-last-24-hours/%7B%7B%20data.link
crawl-data/CC-MAIN-2022-05/segments/1642320300343.4/crawldiagnostics/CC-MAIN-20220117061125-20220117091125-00631.warc.gz
21016412
40062

解决方案：用requests和warcio提取HTML #

通过HTTP范围请求获取WARC文件中指定片段，再用warcio库解析出原始HTML，代码如下：

import requests
from warcio.archiveiterator import ArchiveIterator

# 替换为你从索引中获取的参数
warc_filename = "crawl-data/CC-MAIN-2022-05/segments/1642320303729.69/crawldiagnostics/CC-MAIN-20220122012907-20220122042907-00614.warc.gz"
offset = int("21016412")
length = int("40062")

# Common Crawl WARC文件的基础访问地址
base_url = "https://data.commoncrawl.org/"
warc_url = base_url + warc_filename

# 发送范围请求，仅下载需要的文件片段
headers = {"Range": f"bytes={offset}-{offset + length - 1}"}
response = requests.get(warc_url, headers=headers)

# 解析WARC片段，提取响应内容
for record in ArchiveIterator(response.content):
    if record.rec_type == 'response':
        # 读取原始HTML内容
        html_content = record.content_stream().read()
        # 保存到本地文件
        with open("extracted_page.html", "wb") as f:
            f.write(html_content)
        print("HTML提取完成，已保存到extracted_page.html")
        break

代码说明 #

• 范围请求：通过Range请求头只下载WARC文件中目标片段，避免下载整个大文件，提升效率。
• warcio解析：ArchiveIterator处理WARC格式内容，筛选出response类型记录，其中包含页面的原始HTML。
• 注意事项：如果索引中返回的状态码是404（如示例），提取到的会是网站的404错误页面，而非目标文章内容。

内容的提问来源于stack exchange，提问作者js16