如何用C++17的codecvt_byname实现Latin1转UTF-8用于JSON

问题背景 #

需要处理从PLC获取的Latin1编码字节数组，转换为可用于nlohmann::json的格式并保留Latin1编码，但使用示例代码时出现编译错误，提示~codecvt()和~codecvt_byname()为受保护成员。

编译错误信息 #

/usr/bin/g++   -O3 -DNDEBUG -std=c++17 -MD -MT CMakeFiles/encod.dir/src/encod.cpp.o -MF CMakeFiles/encod.dir/src/encod.cpp.o.d -o CMakeFiles/encod.dir/src/encod.cpp.o -c /src/encod.cpp
In file included from /usr/include/c++/12/locale:43,
                 from /src/encod.cpp:1:
/usr/include/c++/12/bits/locale_conv.h: In instantiation of ‘std::__detail::_Scoped_ptr<_Tp>::~_Scoped_ptr() [with _Tp = std::codecvt<wchar_t, char, __mbstate_t>]’:
/usr/include/c++/12/bits/locale_conv.h:309:7:   required from here
/usr/include/c++/12/bits/locale_conv.h:241:26: error: ‘virtual std::codecvt<wchar_t, char, __mbstate_t>::~codecvt()’ is protected within this context
  241 |         ~_Scoped_ptr() { delete _M_ptr; }
      |                          ^~~~~~~~~~~~~
In file included from /usr/include/c++/12/bits/locale_facets_nonio.h:2067,
                 from /usr/include/c++/12/locale:41:
/usr/include/c++/12/bits/codecvt.h:429:7: note: declared protected here
  429 |       ~codecvt();
      |       ^
In file included from /usr/include/c++/12/memory:76,
                 from /src/encod.cpp:6:
/usr/include/c++/12/bits/unique_ptr.h: In instantiation of ‘void std::default_delete<_Tp>::operator()(_Tp*) const [with _Tp = std::codecvt_byname<wchar_t, char, __mbstate_t>]’:
/usr/include/c++/12/bits/unique_ptr.h:396:17:   required from ‘std::unique_ptr<_Tp, _Dp>::~unique_ptr() [with _Tp = std::codecvt_byname<wchar_t, char, __mbstate_t>; _Dp = std::default_delete<std::codecvt_byname<wchar_t, char, __mbstate_t> >]’
/src/encod.cpp:18:152:   required from here
/usr/include/c++/12/bits/unique_ptr.h:95:9: error: ‘std::codecvt_byname<_InternT, _ExternT, _StateT>::~codecvt_byname() [with _InternT = wchar_t; _ExternT = char; _StateT = __mbstate_t]’ is protected within this context
   95 |         delete __ptr;
      |         ^~~~~~~~~~~~
/usr/include/c++/12/bits/codecvt.h:722:7: note: declared protected here
  722 |       ~codecvt_byname() { }
      |       ^

示例代码 #

#include <locale>
#include <codecvt>
#include <vector>
#include <string>
#include <iostream>
#include <memory>

int main() {
    std::vector<uint8_t> v = {0x68, 0xe4, 0x6c, 0x6c, 0x6f}; // hällo

    std::string my_string(v.begin(), v.end());

    // Convert to wide string
    std::wstring_convert<std::codecvt_utf8<wchar_t>> utf8_conv;
    std::wstring wide_str = utf8_conv.from_bytes(my_string);

    // Convert wide string to Latin1 string
    std::unique_ptr<std::codecvt_byname<wchar_t, char, std::mbstate_t>> 
            latin1_cvt(new std::codecvt_byname<wchar_t, char, std::mbstate_t>("iso-8859-1"));
    std::wstring_convert<std::codecvt<wchar_t, char, std::mbstate_t>> latin1_conv(latin1_cvt.get());
    std::string latin1_str = latin1_conv.to_bytes(wide_str);


    std::cout << latin1_str << std::endl;

    return 0;
}

解决方案 #

一、修复编译错误 #

错误核心是std::codecvt和std::codecvt_byname的析构函数为protected，无法通过默认的unique_ptr删除器或直接delete销毁。以下两种方法可解决：

1. 自定义删除器适配unique_ptr

给unique_ptr提供自定义删除器，通过locale机制确保析构函数可访问：

#include <locale>
#include <codecvt>
#include <vector>
#include <string>
#include <iostream>
#include <memory>

struct CodecvtDeleter {
    template<typename T>
    void operator()(T* ptr) const {
        std::locale().global(std::locale());
        delete ptr;
    }
};

int main() {
    std::vector<uint8_t> v = {0x68, 0xe4, 0x6c, 0x6c, 0x6f}; // hällo
    std::string latin1_str(v.begin(), v.end());

    // 修复后的编码转换逻辑
    std::unique_ptr<std::codecvt_byname<wchar_t, char, std::mbstate_t>, CodecvtDeleter> 
            latin1_cvt(new std::codecvt_byname<wchar_t, char, std::mbstate_t>("iso-8859-1"));
    std::wstring_convert<std::codecvt<wchar_t, char, std::mbstate_t>> latin1_conv(latin1_cvt.get());
    
    std::wstring wide_str = latin1_conv.from_bytes(latin1_str);
    std::string converted_back = latin1_conv.to_bytes(wide_str);

    std::cout << latin1_str << std::endl;
    return 0;
}

2. 通过locale管理codecvt对象

直接创建包含Latin1编码的locale，由locale自动管理codecvt生命周期：

#include <locale>
#include <codecvt>
#include <vector>
#include <string>
#include <iostream>

int main() {
    std::vector<uint8_t> v = {0x68, 0xe4, 0x6c, 0x6c, 0x6f}; // hällo
    std::string latin1_str(v.begin(), v.end());

    // 创建带Latin1编码的locale
    std::locale latin1_locale(std::locale(), new std::codecvt_byname<wchar_t, char, std::mbstate_t>("iso-8859-1"));
    std::wstring_convert<std::codecvt<wchar_t, char, std::mbstate_t>> latin1_conv;
    latin1_conv.imbue(latin1_locale);

    std::wstring wide_str = latin1_conv.from_bytes(latin1_str);
    std::string converted_back = latin1_conv.to_bytes(wide_str);

    std::cout << latin1_str << std::endl;
    return 0;
}

二、nlohmann::json适配要点 #

nlohmann::json默认以UTF-8处理字符串，但如果需要保留Latin1编码：

• 直接存储原始std::string即可，因为std::string本质是字节序列，只要后续读取时按Latin1解码就能还原内容
• 如果需要跨平台兼容JSON，建议将Latin1转换为UTF-8后存储，避免编码歧义

三、是否使用ICU库？ #

• 简单场景：仅需Latin1与宽字符/UTF-8转换，修复后的标准库代码足够，无需引入ICU
• 复杂场景：涉及多语言编码转换、边缘字符处理或追求跨平台稳定性，ICU库更合适——它提供全面的编码支持，且不受C标准库codecvt组件（C17弃用、C++20移除）的兼容性影响

内容的提问来源于stack exchange，提问作者juwalter