PHP大文件下载触发服务器内存告警，咨询多进程下载超50GB文件可行性

Hey there, let's break down your problem step by step and figure out how to fix it!

为什么你的代码会触发内存告警？ #

Your current code uses file_put_contents("10gb.zip", fopen("http://website.website/10GB.zip", 'r')) — here's the core issue:

• By default, fopen() for remote URLs reads the entire file into memory before writing it to disk (unless the remote server supports chunked transfer encoding and PHP is configured to handle it properly).
• Even if you've set memory_limit in php.ini to a high value or -1 (unlimited), your server's physical RAM is the hard cap. When the downloaded data hits ~3.79GB, that's exactly how much free memory your server had available, so the process gets stuck or killed.

大文件下载的常见限制 #

Yes, there are several key limits you need to account for:

• PHP Memory Limit: The memory_limit directive in php.ini controls how much memory a single PHP process can use. Even if you raise it, physical server RAM is a non-negotiable hard cap.
• Execution Time: max_execution_time limits how long a PHP script can run. A 50GB download will take way longer than the default 30 seconds, so you'll need to adjust this or use a CLI script (which doesn't enforce this limit by default).
• Remote Server Support: If the remote server doesn't accept Range HTTP headers (required for chunked/download resuming), you can't split the file into parts for multi-process downloads.

解决方案：低内存单进程下载 #

First, let's fix the single-process download to avoid memory issues entirely. Instead of reading the entire file into memory, we'll read and write in small, manageable chunks:

<?php
$remoteUrl = "http://website.website/10GB.zip";
$localFile = "10gb.zip";
$chunkSize = 1024 * 1024; // 1MB chunks (adjust based on your server's resources)

// Open remote and local file handles
$remoteHandle = fopen($remoteUrl, 'rb');
$localHandle = fopen($localFile, 'wb');

if (!$remoteHandle || !$localHandle) {
    die("Failed to open remote or local file!");
}

// Read and write in chunks to keep memory usage low
while (!feof($remoteHandle)) {
    $chunk = fread($remoteHandle, $chunkSize);
    fwrite($localHandle, $chunk);
}

// Clean up resources
fclose($remoteHandle);
fclose($localHandle);

echo "File downloaded successfully!";
?>

This approach only uses ~1MB of memory at a time, regardless of how large the file is.

实现5进程下载50GB+文件 #

To use multiple processes, you'll need to follow these steps:

1. Verify remote server supports Range requests: Send a HEAD request and check for Accept-Ranges: bytes in the response headers.
2. Split the file into equal parts: For a 50GB file, each process handles 10GB (e.g., process 1 downloads bytes 0-10737418239, process 2 downloads 10737418240-21474836479, etc.).
3. Spawn multiple PHP processes: Use the pcntl extension (only available in CLI mode — web servers don't support multi-process PHP scripts) or run separate CLI scripts for each chunk.
4. Merge the chunks into a single file once all downloads are complete.

Example: Multi-process chunk download (CLI only) #

Here's a simplified outline using pcntl:

<?php
$remoteUrl = "http://website.website/50GB.zip";
$localFile = "50gb.zip";
$totalProcesses = 5;

// Get total file size via HEAD request
$headers = get_headers($remoteUrl, true);
if (!isset($headers['Accept-Ranges']) || $headers['Accept-Ranges'] !== 'bytes') {
    die("Remote server doesn't support range requests — multi-process download isn't possible!");
}
$fileSize = $headers['Content-Length'];
$chunkSize = (int)ceil($fileSize / $totalProcesses);

// Spawn child processes for each chunk
for ($i = 0; $i < $totalProcesses; $i++) {
    $pid = pcntl_fork();
    if ($pid == -1) {
        die("Failed to fork process $i!");
    } elseif ($pid == 0) {
        // Child process: download assigned chunk
        $startByte = $i * $chunkSize;
        $endByte = ($i == $totalProcesses - 1) ? $fileSize - 1 : ($i + 1) * $chunkSize - 1;
        
        $ch = curl_init($remoteUrl);
        curl_setopt($ch, CURLOPT_RANGE, "$startByte-$endByte");
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
        curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
        curl_setopt($ch, CURLOPT_TIMEOUT, 0); // No timeout for long downloads
        
        $chunkData = curl_exec($ch);
        if (curl_errno($ch)) {
            die("Chunk $i download failed: " . curl_error($ch));
        }
        curl_close($ch);
        
        // Write chunk to temporary file
        file_put_contents("chunk_$i.part", $chunkData);
        exit(0); // Exit child process
    }
}

// Wait for all child processes to finish
while (pcntl_waitpid(0, $status) != -1);

// Merge all chunks into the final file
$finalHandle = fopen($localFile, 'wb');
for ($i = 0; $i < $totalProcesses; $i++) {
    $chunkHandle = fopen("chunk_$i.part", 'rb');
    while (!feof($chunkHandle)) {
        fwrite($finalHandle, fread($chunkHandle, 1024 * 1024));
    }
    fclose($chunkHandle);
    unlink("chunk_$i.part"); // Clean up temporary chunk files
}
fclose($finalHandle);

echo "50GB file downloaded successfully with 5 processes!";
?>

Important Notes: #

• CLI Only: Multi-process PHP with pcntl won't work in web environments (Apache/Nginx). You'll need to run this script via the command line.
• Error Handling: Add checks for failed downloads, resume support, and retry logic to make the script production-ready.
• Server Resources: 5 processes will use more bandwidth and CPU, so ensure your server can handle the load.

内容的提问来源于stack exchange，提问作者Sakib Mahmud