大文件自动识别bz2压缩类型并逐行读取的技术问题

Perfect question—handling huge (4GB+) files without loading everything into memory is a critical pain point, and fixing your approach is straightforward once you focus on streaming operations and minimal magic number checks.

Step 1: Fix Magic Number Detection (No Full File Load) #

BZ2 files have a well-defined magic number: the first 3 bytes are always BZh (hex: 0x42 0x5A 0x68). You don't need to read the entire file to check this—just grab the first 3 bytes.

Here's a lightweight detection function:

import os

def is_bz2_file(file_path):
    # Read only the first 3 bytes to check BZ2 magic number
    with open(file_path, 'rb') as f:
        magic_bytes = f.read(3)
    return magic_bytes == b'BZh'

This runs in constant time, regardless of file size—no more loading 4GB into memory just for a check.

Step 2: Stream Line-by-Line (No Full File Load) #

Once you know the file type, use streaming file handlers that read data incrementally instead of all at once:

• For plain text: Use standard open() in text mode (it streams lines by default)
• For BZ2 files: Use bz2.BZ2File() (it's a file-like object that decompresses data on the fly, line by line)

Here's a reusable function to handle both cases:

import bz2

def read_large_file_line_by_line(file_path):
    if is_bz2_file(file_path):
        # BZ2File streams decompressed data incrementally
        with bz2.BZ2File(file_path, 'r') as f:
            for line in f:
                # Decode bytes to string if needed (adjust encoding as required)
                yield line.decode('utf-8').strip()
    else:
        # Standard text file streaming
        with open(file_path, 'r', encoding='utf-8') as f:
            for line in f:
                yield line.strip()

How to Use It: #

for line in read_large_file_line_by_line('huge_file.bz2'):
    # Process each line here—no memory overload!
    print(line)

Why This Works #

• Magic number check: Reads only 3 bytes, so even 10GB files are checked instantly.
• Streaming reading: Both bz2.BZ2File and standard open() iterate over lines without loading the entire file into memory. Memory usage stays low, even for massive files.

Edge Cases to Consider #

• Tiny files: If a file is smaller than 3 bytes, is_bz2_file will return False—treat it as plain text (adjust if you need to handle empty files explicitly).
• Non-UTF-8 encoding: Adjust the decode() or open() encoding parameters to match your file's actual encoding (e.g., latin-1).

内容的提问来源于stack exchange，提问作者mr_beginner