std::regex的区域设置感知程度及UTF-8字符串正则适配疑问

Great question—let’s break this down step by step, since regex locale support in C++ can be tricky, especially across macOS and Windows.

1. How Much Locale Awareness Does std::regex Have? #

• By default, std::regex does not enable locale awareness. Only when you pass the std::regex::collate flag during regex construction will character ranges like [a-b] be influenced by the current locale. For example, in a fr_FR.UTF-8 locale, [a-z] might include accented characters like é or ç.
• std::regex_traits does provide interfaces to interact with locales (like character classification or case mapping), but these features only activate when the collate flag is enabled. The default std::regex_traits<char> binds to the global std::locale, but without collate, it will only use ASCII default rules.
• Note: Different compiler implementations (libstdc++, libc++, MSVC) have subtle differences in locale support. Some may not fully handle non-ASCII character classification even with collate enabled.

2. Using UTF-8 in std::string with POSIX Regex Classes ([:w:], [:punct:]) #

The core issue here is: default std::regex is single-byte character-oriented, while UTF-8 is a multi-byte encoding. This leads to critical problems:

• [:w:]: By default, this class is equivalent to [a-zA-Z0-9_], but for non-ASCII UTF-8 characters (like Chinese, accented letters, or Japanese kana), std::regex will split them into individual bytes and never recognize them as part of [:w:]—even if you set a UTF-8 locale and enable collate, since it doesn’t understand multi-byte sequences as single characters.
• [:punct:]: You mentioned this is less critical, but it’s worth noting: default regex will only recognize ASCII punctuation (!@#$%, etc.). UTF-8 full-width punctuation or special symbols (like ¿, €) will be treated as plain bytes and not categorized as punctuation.

So What’s the Fix for UTF-8? #

If you need to handle UTF-8 regex in cross-platform code, two reliable approaches exist:

• Use wide-character regex (std::wregex):
  • On macOS, wchar_t is 32-bit (UTF-32). Convert your UTF-8 std::string to std::wstring (UTF-32), then use std::wregex. With std::regex::collate and a UTF-8 locale, [:w:] will correctly identify Unicode letters and numbers.
  • On Windows, wchar_t is 16-bit (UTF-16), and std::wregex natively supports UTF-16 characters. Since Windows doesn’t have a default UTF-8 locale, using wide characters avoids this dependency. Set a suitable locale (like "en-US"), and [:w:] will recognize ASCII alphanumerics; some implementations also correctly classify non-ASCII Unicode characters.
• Use a third-party library: Boost.Regex is a mature option that natively supports UTF-8 string regex handling and offers more consistent cross-platform locale support, eliminating manual encoding conversion.

3. Cross-Platform Compatibility (macOS + Windows) #

• macOS: The default locale is usually UTF-8 (e.g., en_US.UTF-8). Using std::wregex with UTF-32 conversion plus std::regex::collate should meet your [:w:] needs.
• Windows: Avoid relying on UTF-8 locales (they’re not supported by all Windows versions). Prioritize std::wregex for UTF-16 strings. If you must convert from UTF-8, ensure correct encoding conversion (using MultiByteToWideChar or a cross-platform library like ICU).
• Explicitly set the locale: Call std::locale::global(std::locale("")); at the start of your code to use the system default locale, but remember Windows’ default locale may not be UTF-8—so the wide-character approach is more reliable.

内容的提问来源于stack exchange，提问作者Clearer