寻求支持Levenshtein阈值过滤单字符非空白差异的文本Diff工具

Hey there, great question! Finding a diff tool that ignores minor text tweaks (like single-character typos or small spelling fixes) without custom rules is tricky, but there are a few solid approaches—both custom scripts and (limited) off-the-shelf options. Here's what I recommend:

1. Custom Python Script with Levenshtein Thresholding #

Rolling a custom script is the most flexible way to enforce your edit distance rule, especially since most off-the-shelf tools don't support this natively. You can combine Python's built-in difflib with a Levenshtein distance library to filter out small changes.

First, install the python-Levenshtein package (for fast distance calculations):

pip install python-Levenshtein

Then use this script to compare your files, only flagging differences where the edit distance exceeds your chosen threshold (e.g., 3 for single-character typos):

import Levenshtein
import difflib

def filter_minor_diffs(file1, file2, threshold=3):
    with open(file1, 'r', encoding='utf-8') as f1, open(file2, 'r', encoding='utf-8') as f2:
        lines1 = [line.strip() for line in f1.readlines()]
        lines2 = [line.strip() for line in f2.readlines()]
    
    # Generate raw line-by-line diffs
    differ = difflib.Differ()
    raw_diffs = list(differ.compare(lines1, lines2))
    
    for line in raw_diffs:
        if line.startswith(('+ ', '- ')):
            # Find the corresponding line from the other version
            match_line = None
            if line.startswith('+ '):
                match_line = next((l for l in raw_diffs if l.startswith('- ') and l[2:] in lines1), None)
            else:
                match_line = next((l for l in raw_diffs if l.startswith('+ ') and l[2:] in lines2), None)
            
            if match_line:
                # Calculate edit distance between the pair
                dist = Levenshtein.distance(line[2:], match_line[2:])
                if dist <= threshold:
                    continue  # Skip minor change
            # Print only significant diffs
            print(line)

# Run with your files
filter_minor_diffs('book_version1.txt', 'book_version2.txt', threshold=3)

2. Post-Processing Google's diff-match-patch Output #

Since you're already familiar with diff-match-patch, you can extend it by filtering its results to ignore small edits. Here's how:

from diff_match_patch import diff_match_patch

dmp = diff_match_patch()
text1 = open('book_version1.txt', 'r', encoding='utf-8').read()
text2 = open('book_version2.txt', 'r', encoding='utf-8').read()

# Generate and clean up raw diffs
diffs = dmp.diff_main(text1, text2)
dmp.diff_cleanupSemantic(diffs)  # Optional: Removes trivial whitespace/formatting changes

# Filter out edits smaller than your threshold
threshold = 3
filtered_diffs = [diff for diff in diffs if abs(len(diff[1])) > threshold]

# Print significant changes
for op, text in filtered_diffs:
    if op == -1:
        print(f'- {text}')
    elif op == 1:
        print(f'+ {text}')

This works on the entire text (not just lines) and skips any small edits that don't meet your threshold.

3. GUI Tools with Plugin Potential #

If you prefer a visual tool:

• Meld: It doesn't have built-in Levenshtein filtering, but you can write a custom Python plugin using its API to intercept diff results and hide minor changes.
• KDiff3: Use its "merge similar lines" feature alongside whitespace ignore, but note it doesn't support direct thresholding. For a semi-general solution, pair it with a pre-processing script to normalize common typos first.

Key Tips #

• Adjust the threshold value based on your needs: 2-3 ignores single-character typos, while 5-6 skips small word swaps.
• For large books, process the text in chunks instead of loading everything into memory to avoid slowdowns.

内容的提问来源于stack exchange，提问作者Chris