使用PHP Curl抓取TripAdvisor利雅得酒店名称时遇问题

Hey there! Scraping TripAdvisor can be tricky because they’ve got solid anti-scraping measures in place. Let’s walk through the most likely issues with your current code and how to fix them:

1. Missing Critical Request Headers #

TripAdvisor checks for browser-like headers to block automated requests. Your current code probably isn’t setting essential ones like User-Agent, Accept, or Referer — without these, the server will likely reject your request or return empty content.

Add these headers to your cURL setup:

$headers = [
    'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Referer: https://www.tripadvisor.com/Hotels-g293984-Riyadh_Riyadh_Province-Hotels.html',
    'Accept-Language: en-US,en;q=0.5'
];

curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

2. Hardcoded Dynamic Parameters #

The plSeed, puid, and slot IDs in your $postfields are dynamic values generated by TripAdvisor’s frontend for each session. Hardcoding them will only work temporarily (if at all) — they’ll expire or become invalid on your next request.

Fix: #

• First, send a GET request to the main Riyadh hotels page to capture these dynamic values from the page’s HTML or embedded scripts. Use simple_html_dom to parse the page and extract the required parameters before making your POST request.
• Alternatively, use your browser’s dev tools (Network tab) to inspect a legitimate POST request, copy the fresh parameters, and test with those first.

3. Incomplete cURL Configuration #

Make sure you’ve set these essential cURL options to mimic a browser properly:

// Return the response instead of printing it directly
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// Follow redirects (TripAdvisor often uses them)
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

// Disable SSL verification temporarily (only for testing; use proper certs in production)
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);

// Enable cookies to maintain session state
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt');

4. Anti-Scraping Blocks #

If you’re still getting blocked, TripAdvisor might have flagged your IP or detected automated behavior:

• Add delays between requests (use sleep(2) or similar) to avoid hitting rate limits.
• Consider using proxy IPs to rotate your request source.
• Avoid making too many requests in a short period — mimic human browsing patterns.

Optimized Example Code #

Here’s a revised version of your code incorporating the above fixes:

require_once(APPPATH."../simple_html_dom.php");

// First, get the main hotels page to capture dynamic parameters
$mainUrl = "https://www.tripadvisor.com/Hotels-g293984-Riyadh_Riyadh_Province-Hotels.html";
$ch = curl_init($mainUrl);

$headers = [
    'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language: en-US,en;q=0.5'
];

curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt');

$mainPage = curl_exec($ch);
curl_close($ch);

// Parse the main page to extract dynamic parameters (adjust selectors based on current HTML)
$html = str_get_html($mainPage);
// Example: Extract slot IDs from embedded scripts (pseudocode)
$scriptContent = $html->find('script[type="application/json"]', 0)->innertext;
$params = json_decode($scriptContent, true);
$slot0 = $params['HOTELS_SLOT_0'] ?? '5c1f95f2-50aa-4c1b-89ea-f957ea5607fd';
$abSlot0 = $params['HOTELS_AB_SLOT_0'] ?? 'eb6ce073-5ea4-4182-ab18-9d656fe9bbc5';

// Now make the POST request with fresh parameters
$postUrl = "https://www.tripadvisor.com/HotelsAjax?action=UPDATE_HOTEL_LIST";
$postfields = array(
    "sl_opp_json" => json_encode(["HOTELS_AB_SLOT_0" => $abSlot0, "HOTELS_SLOT_0" => $slot0]),
    "plSeed" => rand(100000000, 999999999), // Generate random seed or extract from main page
    "showSnippets"=>"false",
    "offset"=>"120",
    "reqNum"=>"2",
    "changeSet"=>"",
    "puid" => uniqid() // Or extract from cookies/main page
);

$ch = curl_init($postUrl);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($postfields));
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt');

$response = curl_exec($ch);

// Check for cURL errors
if(curl_errno($ch)){
    echo 'Curl error: ' . curl_error($ch);
} else {
    // Parse the response to get hotel names
    $responseData = json_decode($response, true);
    if($responseData && isset($responseData['results'])){
        foreach($responseData['results'] as $hotel){
            echo $hotel['name'] . "<br>";
        }
    }
}

curl_close($ch);

A quick note: TripAdvisor’s HTML and API structure changes frequently, so you’ll need to update your selectors/parameter extraction logic periodically. Also, make sure you’re complying with their Terms of Service when scraping.

内容的提问来源于stack exchange，提问作者Lghammadi