C++中实现非ASCII带重音英文字符转对应英文本字符的可靠方案问询

Hey there! Let's break down your problem and fix it properly.

First off, what you're dealing with isn't some "system-specific quirk" — it's the nature of UTF-8 multi-byte characters. Accented letters like é are stored as two bytes in UTF-8 (hex 0xC3 0xA9), which show up as -61 and -87 when interpreted as signed char values. Hardcoding these byte values is totally unreliable; it'll fail for other accented characters (like â or ñ) and can't scale to handle all possible cases.

Here are two solid solutions to handle accent removal correctly:

The core idea is to convert your UTF-8 string to UTF-32 first (where each element represents a full Unicode code point, not a split byte). This lets you work with actual characters instead of fragmented bytes. Then we map accented characters to their plain counterparts, filter out standalone accent marks, and convert back to UTF-8.

Here's a working example:

#include <iostream>
#include <fstream>
#include <string>
#include <locale>
#include <codecvt>
#include <algorithm>

// Helper: Map accented Unicode characters to plain Latin letters
char32_t remove_accents(char32_t c) {
    // Cover common accented characters; extend this list as needed
    switch(c) {
        case U'\u00E9': case U'\u00E8': case U'\u00EA': case U'\u00EB':
            return U'e';
        case U'\u00E0': case U'\u00E1': case U'\u00E2': case U'\u00E4':
            return U'a';
        case U'\u00F9': case U'\u00FA': case U'\u00FB': case U'\u00FC':
            return U'u';
        case U'\u00F2': case U'\u00F3': case U'\u00F4': case U'\u00F6':
            return U'o';
        case U'\u00EC': case U'\u00ED': case U'\u00EE': case U'\u00EF':
            return U'i';
        case U'\u00F1':
            return U'n';
        default:
            // Filter out standalone accent marks (like acute/grave accents)
            if(c >= U'\u0300' && c <= U'\u036F') return U'\0';
            return c;
    }
}

int main(int argc, char** argv) {
    if(argc < 2) {
        std::cerr << "Usage: " << argv[0] << " <input-file-path>" << std::endl;
        return 1;
    }

    std::fstream fin(argv[1], std::ios::in);
    std::string line;

    // Set up converter between UTF-8 and UTF-32
    std::wstring_convert<std::codecvt_utf8<char32_t>, char32_t> utf8_utf32_conv;

    while(std::getline(fin, line)) {
        try {
            // Convert UTF-8 string to UTF-32 (each element is a full Unicode character)
            std::u32string u32_line = utf8_utf32_conv.from_bytes(line);

            // Replace accented chars and filter out standalone accent marks
            std::transform(u32_line.begin(), u32_line.end(), u32_line.begin(), remove_accents);
            u32_line.erase(std::remove(u32_line.begin(), u32_line.end(), U'\0'), u32_line.end());

            // Convert back to UTF-8 and print
            std::string cleaned_line = utf8_utf32_conv.to_bytes(u32_line);
            std::cout << cleaned_line << std::endl;
        } catch(const std::range_error& e) {
            std::cerr << "Skipping invalid UTF-8 line: " << line << std::endl;
        }
    }

    fin.close();
    return 0;
}

If your project can include Boost libraries, Boost.Locale provides out-of-the-box tools for Unicode normalization and accent removal — no need to write a huge character mapping table yourself:

#include <iostream>
#include <fstream>
#include <string>
#include <boost/locale.hpp>

int main(int argc, char** argv) {
    if(argc < 2) {
        std::cerr << "Usage: " << argv[0] << " <input-file-path>" << std::endl;
        return 1;
    }

    // Initialize UTF-8-aware locale
    std::locale loc = boost::locale::generator().generate("en_US.UTF-8");
    std::cout.imbue(loc);

    std::fstream fin(argv[1], std::ios::in);
    std::string line;

    while(std::getline(fin, line)) {
        // One line to convert to plain Latin and remove accents
        std::string cleaned_line = boost::locale::transliterate(
            line, 
            "Latin; NFD; [:Nonspacing Mark:] Remove; NFC"
        );
        std::cout << cleaned_line << std::endl;
    }

    fin.close();
    return 0;
}

Your code operates directly on char values, which splits UTF-8 multi-byte characters into isolated bytes. Hardcoding specific byte values to replace letters and deleting remaining non-ASCII bytes is just guesswork. For example, â (UTF-8 0xC3 0xA2, -61 and -94) would have the -61 byte replaced, leaving -94 to be deleted — resulting in â disappearing entirely. This approach can't cover all accented characters and is extremely fragile.

内容的提问来源于stack exchange，提问作者TriHard8