如何用warcio基于requests.get()非流式响应生成完整WARC记录？

Absolutely! You totally can preserve the payload when using requests.get(...) with stream=False—let’s figure out why this issue happens and how to fix it quickly.

Why does this happen? #

When you set stream=False (which is the default, by the way), Requests loads the entire response content into memory right away. It reads through the raw response stream to populate the response.content attribute, which means the underlying stream gets exhausted.

The WARCWriter from warcio, when you pass it a Requests response object, tries to read the payload directly from that raw stream. If the stream is already empty (which it is in non-streaming mode), warcio can only capture the HTTP headers, not the actual content.

Fixes to save payload in non-streaming mode #

We’ve got two solid approaches here—pick the one that fits your use case:

1. Manually pass the loaded content as payload #

This is the most reliable method, since it uses the content that’s already in memory:

import requests
from warcio.warcwriter import WARCWriter

target_url = "https://example.com"
response = requests.get(target_url, stream=False)

# Write to WARC file
with open("output.warc", "wb") as warc_file:
    writer = WARCWriter(warc_file, gzip=False)
    # Explicitly pass response.content as the payload
    warc_record = writer.create_warc_record(
        target_url,
        "response",
        payload=response.content,
        http_headers=response.headers
    )
    writer.write_record(warc_record)

2. Reset the raw response stream (if supported) #

Some response streams allow you to "rewind" them using seek(0). This lets warcio read the stream again:

import requests
from warcio.warcwriter import WARCWriter

target_url = "https://example.com"
response = requests.get(target_url, stream=False)

# Try to reset the raw stream to the start
if hasattr(response.raw, 'seek'):
    response.raw.seek(0)

with open("output.warc", "wb") as warc_file:
    writer = WARCWriter(warc_file, gzip=False)
    # Now the stream has data again, so we can use the response object directly
    warc_record = writer.create_warc_record(target_url, "response", response=response)
    writer.write_record(warc_record)

A quick note on reliability #

The first method is foolproof because it uses the content that’s already loaded into memory. The second method works only if the response.raw object supports seeking—some streams (like those from certain proxies or non-file-like sources) won’t let you do this, so stick with the first approach if you want consistent results.

内容的提问来源于stack exchange，提问作者kartheek7895