UTF-16与UTF-32的BOM强制使用场景有哪些？

Great question—let’s break down when BOMs (Byte Order Marks) are mandatory for UTF-16 and UTF-32, since this is a super common point of confusion for developers working with non-UTF-8 encodings.

1. Text Files (Disk Storage) #

• For UTF-16 and UTF-32 text files, the BOM is mandatory in most standard contexts if you want the file to be correctly interpreted by common tools (text editors, OS file systems, programming language parsers, etc.).
  • Without a BOM, applications have no reliable way to distinguish between UTF-16LE vs UTF-16BE, or UTF-32LE vs UTF-32BE. For example, Windows’ native text handling relies heavily on BOMs to detect these encodings—skip it, and your file will almost certainly show up as gibberish in Notepad or other system tools.
  • The only exception is if you’ve explicitly agreed on a fixed endianness with every party accessing the file (e.g., a project rule that all UTF-16 files are little-endian without a BOM). But this is non-standard and risky for cross-tool or cross-platform compatibility.

2. C++ In-Memory String Variables #

• Absolutely do NOT include a BOM in C++ string variables like std::u16string or std::u32string.
  • BOMs are a storage/transmission-level marker—their sole purpose is to tell a parser what encoding and endianness to expect when reading bytes from a file or network stream. In-memory strings in C++ are already stored in the system’s native endianness, or handled via standard types that abstract endianness concerns. Adding a BOM here would just be extra, invalid data that your code would have to strip off before processing, leading to bugs (like incorrect string lengths or unexpected characters).
  • For example: If you have a std::u16string containing the word "hello", prepending the BOM (0xFEFF) would make the string length 6 instead of 5, and any string operations (like find or substr) would treat that BOM as a regular character—which it’s not, for in-memory use.

3. Network/Inter-Process Transmission #

• The BOM is mandatory if the encoding and endianness aren’t explicitly agreed upon in advance.
  • If you’re sending UTF-16 or UTF-32 data over a network, between processes, or to an API that doesn’t specify a fixed endianness for these encodings, the BOM is the only reliable way the receiver can correctly parse the byte order.
  • That said, if you have a pre-defined contract (e.g., "all UTF-16 data sent via this API is big-endian"), you can skip the BOM—but only if both sender and receiver strictly adhere to that rule.
  • A quick side note: Many modern APIs and protocols prefer UTF-8 for transmission precisely to avoid these endian/BOM headaches. But if you have to use UTF-16 or UTF-32, the BOM is your safety net when no pre-negotiated standard exists.

Quick Cheat Sheet #

• Files: Mandatory for standard cross-tool compatibility (skip only if fixed endianness is agreed by all parties).
• C++ in-memory variables: Never include a BOM—they’re irrelevant here.
• Transmission: Mandatory if no fixed endianness contract is in place.

内容的提问来源于stack exchange，提问作者user9002947