PHP获取UTF-8编码HTML的<p>元素显示乱码问题求助

问题 #

尝试从指定URL抓取UTF-8编码的HTML文档，提取所有<p>元素内容，但输出的孟加拉语字符显示为乱码（示例：সà§à¦¬à§à¦°à¦¤ বিশà§à¦¬à¦¾à¦¸: ফà§à¦° দà§à¦°à§à¦à¦à¦¨à¦¾à¦° à¦à¦¬à¦²à§ দà§à...）。相关PHP代码如下：

<?php
    error_reporting(E_ALL);
    ini_set('display_errors', 1);
    header('Content-Type: text/plain; charset=utf-8');
    header('Access-Control-Allow-Origin: *');
    header('Access-Control-Allow-Methods: POST, GET, OPTIONS');

    $url = "https://www.sangbadpratidin.in/kolkata/ispat-express-met-an-accident-near-howrah-junction/#.Y7qC6YFeT80.whatsapp";

    $user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36";
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_VERBOSE, true);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_URL,$url);
    $html=curl_exec($ch);

    if (!curl_errno($ch)) {
        $resultStatus = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        if ($resultStatus == 200) {
            @$DOM = new DOMDocument;
            @$DOM->loadHTML($html);
            
            $bodies = $DOM->getElementsByTagName('p');
            foreach($bodies as $body){
                $para = $body->nodeValue;
                echo $para;
            }
        }
    }
?>

原因分析 #

乱码的核心原因是**DOMDocument::loadHTML默认将输入解析为ISO-8859-1编码**，即使原网页是UTF-8编码，也会被错误转码，导致非ASCII字符（如孟加拉语）显示为乱码。此外，curl获取内容时若未明确指定编码处理，也可能加剧编码混乱。

解决方法 #

1. 强制DOMDocument以UTF-8编码解析HTML #

在调用loadHTML前，给HTML内容添加XML声明，让DOMDocument识别正确编码：

// 替换原loadHTML行
$DOM->loadHTML('<?xml encoding="UTF-8">' . $html);

2. 优化curl的编码处理 #

添加CURLOPT_ENCODING选项，让curl自动处理服务器返回的编码（如gzip压缩），确保获取到原始UTF-8内容：

curl_setopt($ch, CURLOPT_ENCODING, ''); // 自动识别并解压编码

完整修改后的代码 #

<?php
    error_reporting(E_ALL);
    ini_set('display_errors', 1);
    header('Content-Type: text/plain; charset=utf-8');
    header('Access-Control-Allow-Origin: *');
    header('Access-Control-Allow-Methods: POST, GET, OPTIONS');

    $url = "https://www.sangbadpratidin.in/kolkata/ispat-express-met-an-accident-near-howrah-junction/#.Y7qC6YFeT80.whatsapp";

    $user_agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36";
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
    curl_setopt($ch, CURLOPT_VERBOSE, true);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_ENCODING, ''); // 添加编码处理
    curl_setopt($ch, CURLOPT_URL,$url);
    $html=curl_exec($ch);

    if (!curl_errno($ch)) {
        $resultStatus = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        if ($resultStatus == 200) {
            $DOM = new DOMDocument;
            // 处理HTML解析警告，替代@错误抑制
            libxml_use_internal_errors(true);
            // 强制以UTF-8解析
            $DOM->loadHTML('<?xml encoding="UTF-8">' . $html);
            libxml_clear_errors();
            
            $bodies = $DOM->getElementsByTagName('p');
            foreach($bodies as $body){
                $para = $body->nodeValue;
                echo $para . "\n\n"; // 添加换行区分段落
            }
        }
    }
?>

额外说明 #

• 移除了@错误抑制符，改用libxml_use_internal_errors(true)处理HTML解析警告，更利于调试。
• 输出时添加换行符，让段落内容更易读。

内容的提问来源于stack exchange，提问作者user21123648