如何将wget的stdout与stderr转换为CSV格式用于分析

I get it—sed can get messy when you need to track state like original URLs and timestamps across multi-line blocks. Awk is perfect for this job because it lets you easily keep track of variables as you process each line. Here's a tailored solution that meets all your conversion requirements:

The Awk Script #

Save this as convert_wget.awk:

BEGIN {
    FS = "--"
    OFS = ","
    original_url = ""
    last_ts = ""
}

/^$/ { next }

/^--[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}--/ {
    split($2, parts, " ")
    ts = parts[1] " " parts[2]
    url = substr($3, 2)
    
    # Only update original URL if this is the start of a new request (not a redirect follow)
    if (prev_line !~ /Location:/) {
        original_url = url
    }
    
    last_ts = ts
    gsub(/,/, ";", url)
    print original_url, last_ts, url
    prev_line = $0
    next
}

{
    gsub(/,/, ";")
    print original_url, last_ts, $0
    prev_line = $0
}

How to Use It #

Run this command against your wget output file:

awk -f convert_wget.awk wget.txt > output.csv

Breakdown of the Logic #

Let’s walk through how this matches your requirements:

1. Comma Replacement: Every line has commas replaced with semicolons using gsub(/,/, ";") to avoid CSV formatting issues.
2. Request Block Separation: Empty lines are skipped entirely, which naturally separates each request’s output block.
3. Original URL Tracking: We only update the original_url variable when we hit a new request start line that isn’t a follow-up to a redirect (checked by looking at the previous line for Location:). This ensures the first column always stays as the initial URL you requested.
4. Timestamp Inheritance: We store the most recent timestamp in last_ts and reuse it for all lines in the same request block until a new timestamp is found.
5. Required Columns: The first column (original URL) and third column (line content) are always present, with the timestamp column included as requested.

Why This Beats Sed #

Sed is great for simple text transformations, but it struggles with stateful tasks like remembering the original URL across multiple lines of redirects. Awk’s built-in variable handling makes it trivial to track this context as we process each line of the wget output.

内容的提问来源于stack exchange，提问作者Christopher Brooks