Log Parser扩展需求：Icecast访问日志解析与PHP文件定制

Hey there! Let’s walk through how to parse your Icecast logs and customize your PHP script with Log Parser support—here’s what you need to know:

First, let’s break down your Icecast log format #

Your logs follow a custom text format, so let’s map each field to make parsing easier. Here’s an example entry:

11.11.111.11 - 5229 [08/May/2018:11:43:38 +0200] "GET /chillout_delicate.ogg HTTP/1.1" 200 36256 "-" "Dalvik/1.6.0 (Linux; U; Android 4.3; GT-I9300 Build/JSS15J)" 0

Each segment corresponds to:

• Client IP: 11.11.111.11
• Username (supports numeric or X/Y formats): 5229 or 2510/14
• Request timestamp: [08/May/2018:11:43:38 +0200]
• Request details: "GET /chillout_delicate.ogg HTTP/1.1"
• HTTP status code: 200
• Bytes sent: 36256
• Referrer: "-" (empty in this case)
• User agent: "Dalvik/1.6.0 (Linux; U; Android 4.3; GT-I9300 Build/JSS15J)"
• Extra flag: 0/1

1. Configure Log Parser to match your log format #

Log Parser doesn’t natively recognize Icecast’s custom format, so you’ll need to use the TEXT input mode and EXTRACT_TOKEN function to pull out each field. Here’s a sample SQL query tailored to your logs:

SELECT 
    EXTRACT_TOKEN(Line, 0, ' ') AS ClientIP,
    EXTRACT_TOKEN(Line, 2, ' ') AS Username,
    TO_TIMESTAMP(EXTRACT_TOKEN(Line, 3, ' [') + ' ' + EXTRACT_TOKEN(Line, 4, ']'), 'dd/MMM/yyyy:HH:mm:ss zzz') AS RequestTime,
    EXTRACT_TOKEN(Line, 5, '"') AS RequestMethod,
    EXTRACT_TOKEN(Line, 6, ' "') AS RequestPath,
    EXTRACT_TOKEN(Line, 7, ' HTTP/') AS HttpVersion,
    EXTRACT_TOKEN(Line, 8, ' ') AS StatusCode,
    EXTRACT_TOKEN(Line, 9, ' ') AS BytesSent,
    EXTRACT_TOKEN(Line, 10, '" "') AS Referrer,
    EXTRACT_TOKEN(Line, 11, '" ') AS UserAgent,
    EXTRACT_TOKEN(Line, 12, ' ') AS ExtraFlag
FROM 'path/to/your/icecast.log'
-- Optional: Filter only audio requests
WHERE Line LIKE 'GET%.ogg%'

• EXTRACT_TOKEN splits each line by the specified delimiter (like spaces or quotes) to grab individual fields.
• TO_TIMESTAMP converts the raw timestamp string into a usable date/time value for sorting/filtering.

2. Call Log Parser from your PHP script #

To integrate this into your PHP file, use shell_exec() or exec() to run the Log Parser command and capture the output. Here’s a working example:

// Define paths (adjust for your OS: Linux uses "logparser", Windows uses the full exe path)
$logFilePath = '/var/log/icecast/access.log';
$logParserPath = 'C:\Program Files\Log Parser 2.2\logparser.exe';

// Build the Log Parser query
$logQuery = <<<SQL
SELECT 
    EXTRACT_TOKEN(Line, 0, ' ') AS ClientIP,
    EXTRACT_TOKEN(Line, 2, ' ') AS Username,
    TO_TIMESTAMP(EXTRACT_TOKEN(Line, 3, ' [') + ' ' + EXTRACT_TOKEN(Line, 4, ']'), 'dd/MMM/yyyy:HH:mm:ss zzz') AS RequestTime,
    EXTRACT_TOKEN(Line, 6, ' "') AS RequestPath,
    EXTRACT_TOKEN(Line, 8, ' ') AS StatusCode,
    EXTRACT_TOKEN(Line, 11, '" ') AS UserAgent
FROM '$logFilePath'
SQL;

// Execute the command and get CSV output
$command = "\"$logParserPath\" -i:TEXT -o:CSV \"$logQuery\"";
$csvOutput = shell_exec($command);

// Convert CSV into a structured PHP array
$rows = explode("\n", trim($csvOutput));
$headers = str_getcsv(array_shift($rows));
$parsedLogs = [];

foreach ($rows as $row) {
    if (empty($row)) continue;
    $parsedLogs[] = array_combine($headers, str_getcsv($row));
}

// Example: Print the first parsed log entry
print_r($parsedLogs[0]);

• Make sure your PHP process has permission to read the log file and execute Log Parser. On Linux, this might mean granting www-data access to the log directory; on Windows, adjust IIS/Apache permissions.

3. Fix common parsing issues #

• Wrong field values: Double-check the index in EXTRACT_TOKEN—remember, indexes start at 0. For example, the username is the 3rd segment (index 2) because the first is IP, second is the hyphen.
• Timestamp conversion errors: Verify the TO_TIMESTAMP format string matches your log’s timestamp. dd/MMM/yyyy:HH:mm:ss zzz works for 08/May/2018:11:43:38 +0200.
• Username format variations: Log Parser will capture both 5229 and 2510/14 as-is. If you need to split the X/Y format, use PHP’s explode('/', $username) after parsing.

4. Alternative: Pure PHP parsing (no Log Parser dependency) #

If you don’t want to rely on external tools, use a regular expression to parse logs directly in PHP. Here’s a regex tailored to your format:

$logLine = '111.111.11.111 - 2510/14 [08/May/2018:11:43:39 +0200] "GET /pub3.ogg HTTP/1.1" 200 36467 "-" "Dalvik/1.6.0 (Linux; U; Android 4.4.2; GT-P5200 Build/KOT49H)" 1';

// Regex pattern to match all fields
$pattern = '/^(\S+) - (\S+) \[([^\]]+)\] "(\S+) (\S+) (\S+)" (\d+) (\d+) "([^"]*)" "([^"]*)" (\d+)$/';
preg_match($pattern, $logLine, $matches);

// Map matches to readable keys
$parsedLog = [
    'client_ip' => $matches[1],
    'username' => $matches[2],
    'request_time' => $matches[3],
    'method' => $matches[4],
    'request_path' => $matches[5],
    'http_version' => $matches[6],
    'status_code' => $matches[7],
    'bytes_sent' => $matches[8],
    'referrer' => $matches[9],
    'user_agent' => $matches[10],
    'extra_flag' => $matches[11]
];

print_r($parsedLog);

This approach keeps everything within PHP, no external tools required.

内容的提问来源于stack exchange，提问作者Hesperson