为何GNU Diff仅支持UTF-8不兼容UTF-16？Git及BOM相关问询

Let’s break down your questions one by one, covering tool design choices, historical context, and platform-specific quirks:

Why does GNU Diff (adopted by Git) only support UTF-8 and not recognize UTF-16? #

GNU Diff was built with traditional text processing in mind—think ASCII and single-byte encodings that align with Unix/Linux ecosystem norms. UTF-8 became the de facto Unicode text encoding because it’s backward-compatible with ASCII: every ASCII character maps directly to a single UTF-8 byte, so tools designed for ASCII work seamlessly with UTF-8.

UTF-16, by contrast, uses 2 or 4 bytes per character, which often includes null bytes (\0) that traditional text tools interpret as markers for binary files. GNU Diff defaults to skipping binary files (since diffing binary data isn’t meaningful in most text workflows), so it treats UTF-16 files as binary rather than text. Git inherits this behavior because it relies on GNU Diff’s core logic for comparing file changes.

Why hasn’t this compatibility issue been fixed? #

It’s less about "not fixing" and more about prioritization and tradeoffs:

• UTF-8 dominates in modern text workflows, especially in the open-source world where Git and GNU tools originated. UTF-16 use cases are far less common for source code and plain text.
• Adding robust UTF-16 support would require complex encoding detection logic (to distinguish UTF-16LE/BE, handle BOMs, etc.), which could introduce bugs or slow down the tool for the majority of users who use UTF-8.
• Workarounds already exist: You can force Git to treat UTF-16 files as text with git diff --text, or configure custom diff tools that support Unicode encodings. Some newer versions of diffutils also have improved Unicode handling, though it’s not enabled by default.

Why do most developers ignore the BOM even though it’s part of the Unicode standard? #

The BOM (Byte Order Mark) serves a purpose, but it’s often skipped for practical reasons:

• UTF-8 doesn’t need it: The Unicode standard allows but discourages BOMs in UTF-8 because they add unnecessary leading bytes that break compatibility with Unix/Linux tools. For example, a shell script starting with #!/bin/bash will fail if preceded by a UTF-8 BOM, since the shebang line becomes invalid.
• UTF-16 byte order assumptions: On Windows, UTF-16LE is the default, so many tools assume little-endian encoding without needing a BOM. Adding a BOM can sometimes cause issues with cross-platform tools that don’t expect it.
• Historical habit: Most developers cut their teeth on ASCII-based tools that had no concept of BOMs. Avoiding BOMs feels familiar and reduces the risk of unexpected issues across different environments.

Do UTF-16 source files on Windows affect tool usage? #

Absolutely—here are the most common pain points:

• Git will flag UTF-16 files as binary by default, so git diff won’t show line-by-line changes (it’ll just say "binary files differ").
• Cross-platform text editors and build tools (like Make, CMake, or even some IDEs) may display UTF-16 files as gibberish if they default to UTF-8.
• Scripts written in UTF-16 will fail to run on Unix/Linux systems, which expect UTF-8 or ASCII.

To mitigate this, most developers either:

• Convert UTF-16 files to UTF-8 (preferred for cross-platform compatibility),
• Configure Git to handle UTF-16 via .gitattributes (e.g., *.cs diff=utf16 tells Git to convert the file to UTF-8 before diffing),
• Use tools explicitly designed to support UTF-16 (like Visual Studio or VS Code, which auto-detect encoding).

内容的提问来源于stack exchange，提问作者Keepun