为何Linux wc -l命令与Python open()读取news-commentary-v12语料文件行数不一致？

Great question! The mismatch in line counts boils down to how wc -l and Python's text-mode file handling interpret line endings, especially when your corpus contains mixed or non-standard newline characters. Let’s break this down clearly:

1. How wc -l Counts Lines #

The wc -l command works directly on the file’s raw byte stream. It counts the number of \n (ASCII 0x0A, line feed) characters. If the file ends without a trailing \n, it still adds 1 to the count for the final line. In short:

wc -l line count = number of '\n' bytes + (1 if file is non-empty and no trailing '\n' else 0)

It completely ignores other line-ending characters like \r (carriage return)—they don’t trigger a new line count.

2. Python Text-Mode ('r') Behavior #

When you open a file in text mode with the default newline=None, Python enables universal newline mode. This means it treats all of the following sequences as valid line endings:

• \n (line feed)
• \r\n (carriage return + line feed)
• \r (carriage return alone)

Python then converts all these line endings to \n in the returned strings. So if your corpus has any standalone \r characters, Python will split those positions into separate lines—resulting in more lines than what wc -l counts.

For example, if a line in your file looks like hello\rworld, wc -l counts this as 1 line (no \n present), but Python’s text-mode readlines() will split it into two lines: hello\n and world.

3. Binary-Mode ('rb') Fixes the Mismatch #

When you open the file in binary mode:

• Python skips all newline translation and universal newline parsing.
• readlines() splits the byte stream only on b'\n' bytes, which matches exactly how wc -l counts lines.
• Decoding the bytes to UTF-8 afterward doesn’t change the line structure—it just converts byte strings to Unicode strings without altering where the splits occurred.

How to Get Matching Line Counts in Text Mode #

If you prefer using text mode and still want to match wc -l’s count, explicitly set newline='\n' when opening the file. This tells Python to only recognize \n as a line ending, ignoring standalone \r characters and aligning with wc -l’s logic:

with open(en_path, 'r', encoding='utf-8', newline='\n') as fp:
    en_lines = fp.readlines()
print(f"en line count: {len(en_lines)}")

内容的提问来源于stack exchange，提问作者Wenbin Lai