PHP获取外部数据：内容超长处理及XPath缺失div的问题求助

Hey Fiona, let's tackle your two PHP/scraping questions with practical, battle-tested solutions—since you're building a learning/tracking site, these should help make your setup more robust and efficient:

1. Handling Large External Content When It Exceeds Your Threshold X #

When fetching external data that might blow past your size limit X, the key is to avoid loading the entire payload into memory at once (which can cause memory exhaustion) or handle truncation safely. Here are your best options:

• Stream-based Chunked Reading
  Use PHP's stream functions to read the content in chunks of your X size, processing each piece incrementally. This is perfect for large files or responses where you don't need the full content at once:

  $remoteUrl = "https://your-target-site.com/large-data";
  $chunkSize = 1024 * 1024; // Adjust to your X value (e.g., 1MB chunks)
  $outputHandle = fopen('local-storage.txt', 'w');
  $remoteHandle = fopen($remoteUrl, 'r');
  
  if ($remoteHandle && $outputHandle) {
      while (!feof($remoteHandle)) {
          $chunk = fread($remoteHandle, $chunkSize);
          // Process or store the chunk here (e.g., append to a file)
          fwrite($outputHandle, $chunk);
      }
      fclose($remoteHandle);
      fclose($outputHandle);
  }
  

• CURL with Custom Buffer Control
  For more control (like handling HTTP headers or timeouts), use CURL with a custom write function to process chunks as they arrive:

  $curl = curl_init($remoteUrl);
  $chunkSize = 1024 * 1024;
  $outputFile = fopen('local-storage.txt', 'w');
  
  curl_setopt($curl, CURLOPT_BUFFERSIZE, $chunkSize);
  curl_setopt($curl, CURLOPT_WRITEFUNCTION, function($curl, $data) use ($outputFile) {
      return fwrite($outputFile, $data);
  });
  curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true); // Handle redirects
  curl_setopt($curl, CURLOPT_TIMEOUT, 30); // Prevent hanging requests
  
  curl_exec($curl);
  curl_close($curl);
  fclose($outputFile);
  

• Safe Truncation (If You Only Need the First X Bytes)
  If you must load the full content but need to cap it at X, use mb_strcut to avoid breaking multi-byte characters (like UTF-8):

  $content = file_get_contents($remoteUrl);
  $maxLength = $X; // Your threshold value
  
  if (mb_strlen($content, 'UTF-8') > $maxLength) {
      $content = mb_strcut($content, 0, $maxLength, 'UTF-8');
      $content .= '[TRUNCATED]'; // Add a marker to indicate truncation
  }
  

• Tune Memory/Time Limits
  If you need to process the full large content, temporarily increase PHP's memory limit for the script:

  ini_set('memory_limit', '256M'); // Adjust based on your needs
  

  Also set timeouts in CURL or stream contexts to avoid long-running requests.

2. Robust XPath Extraction for Bulk Page Processing #

Your current setup works most of the time, but we can make it more fault-tolerant and efficient for bulk tasks:

• Defensive XPath Checks
  Never assume the target div exists—always validate first. Use count() to check for node presence before trying to extract data (compatible with XPath 1.0, which PHP's DOMXPath uses):

  $dom = new DOMDocument();
  libxml_use_internal_errors(true); // Suppress messy HTML parsing warnings
  $dom->loadHTML($htmlContent);
  libxml_clear_errors();
  
  $xpath = new DOMXPath($dom);
  $targetDivCount = $xpath->evaluate('count(//div[@class="your-target-class"])');
  
  if ($targetDivCount > 0) {
      $targetDiv = $xpath->query('//div[@class="your-target-class"]')->item(0);
      $extractedData = trim($targetDiv->nodeValue);
  } else {
      // Handle missing div: log the URL, skip it, or mark for retry
      error_log("Target div missing for URL: $currentUrl");
      $extractedData = null;
  }
  

• Refine Your XPath for Stability
  Avoid fragile absolute paths (like /html/body/div[3]/div[2])—use stable attributes instead:

  • Prefer IDs if available: //div[@id="unique-target-id"]
  • Combine attributes for precision: //div[@class="content-block" and @data-tracking="true"]
  • Use contains() for flexible class matching (if classes might change slightly): //div[contains(@class, "target-content")]

• Bulk Processing Optimizations

  • Retry Logic for Failures
    Sometimes missing divs are due to temporary network glitches. Add a simple retry loop for failed requests:

    function extractWithRetry($url, $retries = 2) {
        // Fetch HTML and run XPath checks here...
        if ($targetDivCount === 0 && $retries > 0) {
            sleep(1); // Avoid hammering the target site
            return extractWithRetry($url, $retries - 1);
        }
        return $extractedData;
    }
    

  • Queue-based Processing
    Store your list of URLs in a database queue (with status columns: pending, completed, failed). Process one URL at a time, updating its status as you go. This lets you resume bulk tasks if your script crashes, and easily revisit failed URLs later.
  • Rate Limiting
    Add small delays between requests (e.g., sleep(1) or random delays like sleep(rand(1,3))) to avoid triggering the target site's anti-scraping measures.

• Logging & Debugging
  Keep detailed logs of every URL processed: note success/failure, timestamp, and error messages. This helps you spot patterns (e.g., a specific section of URLs always fails) and update your XPath if the target site's structure changes.

内容的提问来源于stack exchange，提问作者Fiona Stone