CentOS7服务器上PHP Curl多次请求后失效问题求助

Hey there, let's dig into why your Curl requests are getting redirected to error pages after a few tries—even when you rebuild your Droplet or restore snapshots. This is almost always tied to anti-scraping measures on the target site, so let's break down fixes step by step:

1. Fix Your Request Headers to Mimic a Real Browser #

Most sites flag requests that don't look like they're coming from a regular web browser. Your initial working requests probably slipped through, but repeated bare-bones Curl calls triggered their anti-bot systems.

Update your Curl setup with these critical headers and settings:

$ch = curl_init('https://your-target-page.com');

// Mimic a real browser's user agent (pick one or randomize)
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36');

// Add common request headers
curl_setopt($ch, CURLOPT_HTTPHEADER, [
    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language: en-US,en;q=0.5',
    'Referer: https://your-target-page.com/' // Match the target site's domain
]);

// Enable cookie persistence to maintain session state
curl_setopt($ch, CURLOPT_COOKIEJAR, '/tmp/cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, '/tmp/cookies.txt');

// Follow redirects properly
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 5);

$response = curl_exec($ch);
curl_close($ch);

2. Slow Down Your Request Frequency #

Firing off requests in quick succession is a dead giveaway for bots. Add a delay between requests to mimic human behavior:

// After each successful request
sleep(2); // Wait 2 seconds before the next call

You can even randomize the delay (e.g., sleep(rand(1,3));) to make it look more natural.

3. Check if Your Droplet's IP Is Blacklisted #

DigitalOcean reuses IP addresses across Droplets, and the target site might have blocked the IP your Droplet is using—even if you rebuild it (since you might get the same or a neighboring IP in the same data center).

• Test this: Try running your Curl code from your local machine (or a different network). If it works there but not on your Droplet, the IP is likely blocked.
• Fix it:
  • Switch your Droplet to a different DigitalOcean region.
  • Assign a new Elastic IP to your Droplet (this lets you swap out the IP without rebuilding the whole server).

4. Debug with Curl Verbose Logs #

To get to the root of the redirect, enable verbose logging in Curl to see exactly what's happening with each request:

$ch = curl_init('https://your-target-page.com');
// ... your existing Curl options ...

// Enable verbose logging
curl_setopt($ch, CURLOPT_VERBOSE, true);
$verboseLog = fopen('php://temp', 'w+');
curl_setopt($ch, CURLOPT_STDERR, $verboseLog);

$response = curl_exec($ch);

// Print the verbose log
rewind($verboseLog);
echo "Request Details:\n" . stream_get_contents($verboseLog);
fclose($verboseLog);

// Check the HTTP status code
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
echo "\nHTTP Status Code: " . $httpCode;

curl_close($ch);

Look for:

• A 403 Forbidden status: Confirmation you're being blocked by anti-scraping.
• The redirect target URL: If it's a captcha page or an error page, that's clear evidence of bot detection.

5. Randomize Request Patterns #

If your requests are identical every time, the site's anti-bot system will catch on fast. Try randomizing small details:

• Rotate between multiple user agents (create an array of UAs and pick one at random).
• Adjust the order of your request headers (some sites check for header order consistency).

内容的提问来源于stack exchange，提问作者Trix