使用Python解码text.ucs文件遇UnicodeDecodeError，求解决方案

Let’s break down the issues with your current approach and fix this step by step:

1. You’re opening the file in text mode (not binary) #

In Python 2, using open('text.ucs', 'r') reads the file as plain text, which can mangle binary-encoded data like UCS/UTF-16 by automatically handling newline conversions. You must use binary mode ('rb') to preserve the raw byte sequence of the file.

2. UTF-16 depends on correct byte order handling #

UCS files typically use UTF-16 (either UCS-2 or UCS-4), a 16-bit encoding that relies on byte order (endianness). When you call decode('utf-16'), Python looks for a Byte Order Mark (BOM: \xff\xfe for little-endian, \xfe\xff for big-endian) to determine the order. If there’s no BOM, it defaults to big-endian—this mismatch is likely causing your decoding error.

Step-by-Step Solution #

First, read the file correctly in binary mode

Replace your initial code with this to capture the raw, unmodified bytes:

# Open in binary mode to preserve the original byte data
with open('text.ucs', 'rb') as file:
    raw_bytes = file.read()

# Print the raw byte representation to inspect what you’re working with
print(repr(raw_bytes))

Then, test different UTF-16 variants for decoding

Try these options one by one to find the correct encoding for your file:

# Option 1: Auto-detect BOM (default utf-16)
try:
    decoded_text = raw_bytes.decode('utf-16')
    print("Decoded with utf-16:", decoded_text)
except UnicodeDecodeError:
    # Option 2: Try little-endian UTF-16 (common on most modern systems)
    try:
        decoded_text = raw_bytes.decode('utf-16-le')
        print("Decoded with utf-16-le:", decoded_text)
    except UnicodeDecodeError:
        # Option 3: Try big-endian UTF-16
        decoded_text = raw_bytes.decode('utf-16-be')
        print("Decoded with utf-16-be:", decoded_text)

Why your original code failed #

• Using text mode ('r') altered the raw byte data, breaking the encoding structure needed for proper decoding.
• Without a BOM in the file or explicit byte order specification, utf-16 decoding assumed the wrong endianness, triggering the UnicodeDecodeError.

内容的提问来源于stack exchange，提问作者cyborg