批量清理同内容短TXT文件：二进制对比实现方案咨询

Alright, let's break down how to solve this problem exactly as you described—handling thousands of TXT files, using pairwise binary comparisons to delete shorter files that are prefixes of longer ones. Here's a practical, actionable implementation plan with code you can use right away:

First, let's lock in the rule we're following: For any pair of files, if one is a perfect binary prefix of the other (and shorter), we delete the shorter file. We'll use the n*(n-1) pairwise comparison approach, but we'll add checks to avoid redundant work (like rechecking files we already plan to delete).

Python is ideal here because it handles binary file I/O smoothly, works cross-platform, and makes it easy to manage file metadata.

1. Gather File Metadata & Binary Content #

First, we'll scan the target folder and collect each TXT file's path and full binary content. Reading in binary mode ensures we avoid encoding errors that could break our prefix checks.

import os

def collect_file_data(folder_path):
    file_records = []
    for filename in os.listdir(folder_path):
        if filename.lower().endswith('.txt'):
            file_path = os.path.join(folder_path, filename)
            # Read content in binary to preserve exact bytes
            with open(file_path, 'rb') as f:
                binary_content = f.read()
            file_records.append( (file_path, binary_content) )
    return file_records

2. Pairwise Binary Comparison & Mark Files for Deletion #

We'll loop through every pair of files, check if one is a prefix of the other, and mark the shorter one for deletion. We'll skip any files already marked to avoid unnecessary checks.

def identify_files_to_delete(file_records):
    to_delete = set()
    total_files = len(file_records)

    for i in range(total_files):
        path_i, content_i = file_records[i]
        # Skip if we already plan to delete this file
        if path_i in to_delete:
            continue
        len_i = len(content_i)

        for j in range(total_files):
            if i == j:
                continue
            path_j, content_j = file_records[j]
            if path_j in to_delete:
                continue
            
            len_j = len(content_j)
            # Check if file i is a shorter prefix of file j
            if len_i < len_j and content_j.startswith(content_i):
                to_delete.add(path_i)
                break  # No need to check other files once we find a match
            # Check if file j is a shorter prefix of file i
            elif len_j < len_i and content_i.startswith(content_j):
                to_delete.add(path_j)
    
    return to_delete

3. Execute Deletion (With Safety Checks) #

Before deleting anything, we'll print the list of files to remove so you can verify it's correct. Always back up your files first!

def delete_marked_files(to_delete):
    print(f"Found {len(to_delete)} files to delete:\n")
    for file_path in to_delete:
        print(f" - {os.path.basename(file_path)}")
    
    # Uncomment the lines below to actually delete files (after verifying!)
    # print("\nDeleting files...")
    # for file_path in to_delete:
    #     os.remove(file_path)
    # print("Deletion complete.")

4. Run the Full Workflow #

Replace the folder path with your actual TXT folder location, then run the script:

if __name__ == "__main__":
    target_folder = "./your_txt_files_folder"  # Update this path!
    file_data = collect_file_data(target_folder)
    files_to_remove = identify_files_to_delete(file_data)
    delete_marked_files(files_to_remove)

• Performance Warning: The n*(n-1) approach will get slow with thousands of files (e.g., 5,000 files = 25 million comparisons). A quick optimization: sort files by size first, then only compare smaller files to larger ones. Once a small file is marked as a prefix of a larger one, you can stop checking it against other big files.
• Binary Accuracy: Using startswith() on binary strings ensures we're comparing exact byte values—no encoding-related mismatches.
• Duplicate Handling: If two files have identical size and content, the script won't mark either for deletion. To handle duplicates, add a step to group files by their binary content hash and keep only one copy per hash.
• Backup First: Always make a copy of your folder before running deletion code—mistakes happen!

内容的提问来源于stack exchange，提问作者pbies