PHP(CakePHP 2.x)批量检测远程图片异常及Shopify防盗链问题求助

Let’s break down how to fix your inconsistent image detection issues caused by Shopify’s anti-hotlinking and rate-limiting policies. These solutions will make your checks more reliable while respecting Shopify’s constraints.

1. Fix Anti-Hotlinking with Proper Request Headers #

Shopify’s anti-hotlinking system typically checks the Referer and User-Agent headers to block scripted requests. By mimicking a legitimate browser request, you can bypass this restriction. Replace your existing checkRemoteFile function with this improved version:

function checkRemoteFile($url, $retryCount = 3) {
    $ch = curl_init();
    
    // Mimic a browser request to avoid anti-hotlinking blocks
    $headers = [
        'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
        'Referer: ' . parse_url($url, PHP_URL_SCHEME) . '://' . parse_url($url, PHP_URL_HOST)
    ];
    
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_NOBODY, 1); // Only fetch headers, not content
    curl_setopt($ch, CURLOPT_FAILONERROR, 1);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); // Follow redirects if the image is moved
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
    curl_setopt($ch, CURLOPT_TIMEOUT, 10);
    
    $result = curl_exec($ch);
    $httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
    curl_close($ch);
    
    // Handle rate limits (429) or temporary 403 bans with retries
    if (($httpCode === 429 || $httpCode === 403) && $retryCount > 0) {
        // Exponential backoff: wait longer each retry to avoid hitting limits
        $backoffDelay = pow(2, 3 - $retryCount);
        sleep($backoffDelay);
        return checkRemoteFile($url, $retryCount - 1);
    }
    
    return $result !== FALSE;
}

Key improvements here:

• Valid Referer header: Uses the image’s own domain to trick Shopify into thinking the request comes from a legitimate source.
• Browser-like User-Agent: Avoids being flagged as a bot.
• Exponential backoff: Automatically retries failed requests with increasing delays to handle rate limits.

2. Add Caching to Reduce Unnecessary Requests #

Rechecking the same image URLs repeatedly is a major cause of rate limits. Implement caching to store the results of your checks for a reasonable period (e.g., 1 day). Using CakePHP’s built-in Cache component:

function checkRemoteFileWithCache($url) {
    $cacheKey = 'image_validity_' . md5($url);
    
    // Return cached result if available
    if ($isValid = Cache::read($cacheKey)) {
        return $isValid;
    }
    
    // Perform the check and cache the result
    $isValid = checkRemoteFile($url);
    Cache::write($cacheKey, $isValid, '+1 day');
    
    return $isValid;
}

3. Optimize Your Loop Logic #

Add URL validation upfront to skip invalid URLs before making any requests, further reducing unnecessary traffic:

foreach($inventories as $inventory) {
    $imageUrl = $inventory['Inventory']['imgsrc1'];
    
    // Skip invalid URLs immediately
    if (!filter_var($imageUrl, FILTER_VALIDATE_URL)) {
        echo "<p>Invalid URL: ".$inventory['Inventory']['sku']."</p>";
        continue;
    }
    
    if (!checkRemoteFileWithCache($imageUrl)) {
        echo "<p>Pass: ".$inventory['Inventory']['sku']." <a href='". $imageUrl ."'>IMG Here</a> </p>";
        continue;
    }
    
    echo "<p>Product SKU:".$inventory['Inventory']['sku']."</p>";
    // Execute your upload logic here
}

4. Bonus: Use Shopify’s API (If Possible) #

If you have access to Shopify’s API, fetching image details directly via authenticated API requests is the most reliable approach. API requests are exempt from anti-hotlinking restrictions and have clearer rate-limit guidelines. For example, you could use the Products API to retrieve images for a product by its SKU instead of relying on direct image URLs.

内容的提问来源于stack exchange，提问作者Jurgen Feuchter