升级至PHP8.1后通用DOM爬虫（DOMDocument+cURL）报错排查

将运行多年的通用DOM爬虫代码从PHP7.4升级至PHP8.1后，出现如下报错：

DOMDocument::loadHTML(): Argument #1 ($source) must not be empty

代码部署在Joomla 3.10、Apache、Linux环境中，CLI定时任务运行正常，但浏览器环境下失效，且确认变量并非空值。爬虫代码如下：

// acquire data
$source = 'https://www.somesite.com/category/somepage';
$scraper = new DOMScraper();
$scraper->setSite($source)->setSource();
echo $scraper->getInnerHTML('table');

/**
 * Generic DOM scraper using DOMDocument and cURL
 */
Class DOMScraper extends DOMDocument{
    public $site;
    private $source;
    private $dom;

    function __construct(){
        libxml_use_internal_errors(true);
        $this->preserveWhiteSpace = false;
        $this->strictErrorChecking = false;
    }

    function setSite($site){
        $this->site = $site;
        return $this;
    }

    function setSource(){
        if(empty($this->site))return 'Error: Missing $this->site, use setSite() first';
        $this->source = $this->get_data($this->site);
        return $this;
    }

    function getInnerHTML($tag, $id=null, $nodeValue = false){
        if(empty($this->site))return 'Error: Missing $this->source, use setSource() first';
        $this->loadHTML($this->source);
        $tmp = $this->getElementsByTagName($tag);
        $ret = null;
        foreach ($tmp as $v){
            if($id !== null){
                $attr = explode('=',$id);
                if($v->getAttribute($attr[0])==$attr[1]){
                    if($nodeValue == true){
                        $ret .= trim($v->nodeValue);
                    }else{
                        $ret .= $this->innerHTML($v);
                    }
                }
            }else{
                if($nodeValue == true){
                    $ret .= trim($v->nodeValue);
                }else{
                    $ret .= $this->innerHTML($v);
                }
            }
        }
        return $ret;
    }

    function innerHTML($dom){
        $ret = "";
        $nodes = $dom->childNodes;
        foreach($nodes as $v){
            $tmp = new DOMDocument();
            $tmp->appendChild($tmp->importNode($v, true));
            $ret .= trim($tmp->saveHTML());
        }
        return $ret;
    }

    function get_data($url){
        if(function_exists('curl_init')){
            $ch = curl_init();
            curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
            curl_setopt($ch, CURLOPT_TIMEOUT, 5);
            curl_setopt($ch, CURLOPT_URL, $url);
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
            $data = curl_exec($ch);
            curl_close($ch);
            return $data;
        }else{
            return file_get_contents($url);
        }
    }
}

1. PHP8.1严格参数校验：PHP8.0起对函数参数的类型检查更严格，DOMDocument::loadHTML()在PHP7.4中接受空值仅触发警告，但PHP8.1直接抛出错误。
2. 请求来源限制：目标网站可能校验请求的User-Agent，CLI环境下curl默认User-Agent与浏览器环境差异大，导致网站拒绝请求，返回空内容。
3. Joomla环境限制：浏览器环境下Joomla的安全插件（如防火墙、请求过滤）可能拦截爬虫请求；或PHP配置中allow_url_fopen被禁用，导致file_get_contents无法正常获取内容。
4. 代码逻辑错误：getInnerHTML方法错误地检查empty($this->site)而非empty($this->source)，即便$this->source为空也会执行loadHTML。

1. 修复代码逻辑漏洞 #

修改getInnerHTML的校验条件，确保检查的是$this->source：

function getInnerHTML($tag, $id=null, $nodeValue = false){
    if(empty($this->source))return 'Error: Missing $this->source, use setSource() first';
    $this->loadHTML($this->source);
    // 剩余代码保持不变
}

2. 增强请求伪装与错误捕获 #

在get_data方法中添加浏览器User-Agent，并捕获请求错误：

function get_data($url){
    if(function_exists('curl_init')){
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
        curl_setopt($ch, CURLOPT_TIMEOUT, 10); // 延长超时时间提升成功率
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        // 模拟浏览器User-Agent
        curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36');
        // 捕获curl错误并记录日志
        $data = curl_exec($ch);
        if(curl_errno($ch)){
            error_log('Curl请求错误: ' . curl_error($ch));
            curl_close($ch);
            return false;
        }
        curl_close($ch);
        return $data;
    }else{
        // 为file_get_contents添加浏览器请求头
        $context = stream_context_create([
            'http' => [
                'header' => "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36\r\n"
            ]
        ]);
        $data = file_get_contents($url, false, $context);
        if($data === false){
            error_log('file_get_contents获取内容失败: ' . $url);
            return false;
        }
        return $data;
    }
}

3. 增加内容获取校验 #

在setSource方法中检查get_data的返回值，确保$this->source不为空：

function setSource(){
    if(empty($this->site))return 'Error: Missing $this->site, use setSite() first';
    $this->source = $this->get_data($this->site);
    if(empty($this->source)){
        error_log('获取网站内容失败: ' . $this->site);
        return 'Error: Failed to fetch content from site';
    }
    return $this;
}

4. 检查环境配置 #

• 确认Joomla安全插件（如Admin Tools）未阻止对外请求；
• 检查PHP配置中allow_url_fopen是否开启（针对file_get_contents）；
• 确认服务器防火墙未限制对外HTTP请求。

内容的提问来源于stack exchange，提问作者Emiel