如何在Node.js与浏览器环境中可靠解码与转义文本？

问题背景 #

解析二进制文件提取文本时，带重音的字母出现编码乱码，例如得到Señor而非Señor，salvación而非salvación。现有Node.js方案依赖querystring模块的unescape方法，但浏览器环境无法使用该模块，且尝试decodeURI/decodeURIComponent等方法时，要么无法兼顾重音字符与URL编码字符，要么因非法字符抛出错误。

现有Node.js实现代码 #

import { unescape } from 'querystring';

export class CharHelpers {
  cleanString(str: string): string {
    //decode character to UTF-8. Helps with accented characters.
    //This converts things like 'salvación' to 'salvación'
    return (
      unescape(this.escapeString(str))
        //Replace CRLF with a single \n
        .replace(/\r\n/g, '\n')
        //remove any trailing whitespace or newlines
        .trim()
    );
  }

  private escapeString(str: string): string {
    //A replacement for the deprecated escape method
    const allowed = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789@*_+-./,';
    str = str.toString();
    const len = str.length;
    let R = '';
    let k = 0;
    let S = '';
    let chr = '';
    let ord = 0;
    while (k < len) {
      chr = str[k];
      if (allowed.indexOf(chr) !== -1) {
        S = chr;
      } else {
        ord = str.charCodeAt(k);
        if (ord < 256) {
          S = '%' + ('00' + ord.toString(16)).toUpperCase().slice(-2);
        } else {
          S = '%u' + ('0000' + ord.toString(16)).toUpperCase().slice(-4);
        }
      }
      R += S;
      k++;
    }
    return R;
  }
}

测试代码及问题 #

尝试多种解码方法均无法满足需求：

function escapeString(str) {
  const allowed = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789@*_+-./,';
  str = str.toString();
  const len = str.length;
  let R = '';
  let k = 0;
  let S = '';
  let chr = '';
  let ord = 0;
  while (k < len) {
    chr = str[k];
    if (allowed.indexOf(chr) !== -1) {
      S = chr;
    } else {
      ord = str.charCodeAt(k);
      if (ord < 256) {
        S = '%' + ('00' + ord.toString(16)).toUpperCase().slice(-2);
      } else {
        S = '%u' + ('0000' + ord.toString(16)).toUpperCase().slice(-4);
      }
    }
    R += S;
    k++;
  }
  return R;
}

function tryParse(method, s){
  try{
    console.log(method.name , ' -> ', method(s));
  } catch(ex){
    console.error(method.name , ' -> ', ex.message)
  }
}

//-------------------------------

const str = 'Señor and salvación and Number%3A 1234%3B %06%88';
//Expecting: 'Señor and salvación and Number: 1234; ♠'

tryParse(unescape, str); //converts : and ; but not accented chars
tryParse(unescape, escapeString(str)); //converts nothing
tryParse(decodeURI, str); //Malformed URI error
tryParse(decodeURIComponent, str); //Malformed URI error
tryParse(decodeURI, escapeString(str)); //converts accented chars but not : and ;
tryParse(decodeURIComponent, escapeString(str)); //converts accented chars but not : and ;

跨环境解决方案 #

核心思路：乱码本质是UTF-8字节被错误解析为单字节编码（如ISO-8859-1），通过TextEncoder/TextDecoder转回正确编码；同时安全处理URL编码字符，避免报错。

export class CharHelpers {
  cleanString(str: string): string {
    // 修复UTF-8编码错误，处理URL编码字符，再格式化文本
    return this.safeDecodeURIComponent(this.fixUtf8Encoding(str))
      .replace(/\r\n/g, '\n')
      .trim();
  }

  private fixUtf8Encoding(str: string): string {
    // 将被错误解析为ISO-8859-1的字符串转回UTF-8
    const encoder = new TextEncoder('iso-8859-1');
    const decoder = new TextDecoder('utf-8');
    return decoder.decode(encoder.encode(str));
  }

  private safeDecodeURIComponent(str: string): string {
    // 安全解码URL编码字符，非法编码保留原内容
    return str.replace(/%([0-9A-Fa-f]{2})/g, (match) => {
      try {
        return decodeURIComponent(match);
      } catch {
        return match;
      }
    });
  }
}

方案说明 #

• fixUtf8Encoding：解决重音字符乱码问题，比如Señor会被正确转为Señor。
• safeDecodeURIComponent：处理%3A、%3B这类URL编码字符，同时捕获非法编码的异常，避免整个方法崩溃。
• 跨环境支持：TextEncoder/TextDecoder在现代浏览器和Node.js 11+原生支持，无需额外依赖。

测试验证 #

输入测试字符串'Señor and salvación and Number%3A 1234%3B %06%88'，会输出'Señor and salvación and Number: 1234; %06%88'（%06%88为非法编码，保留原内容符合预期）。

内容的提问来源于stack exchange，提问作者Chris Barr