Python Datacompy：对比大文件时如何获取完整差异报告

Yep, I’ve dealt with this exact frustration before—Datacompy’s default 10-row limit for difference displays is pretty useless when you’re working with huge datasets. Luckily, there are a few straightforward ways to get the full picture:

1. Adjust the display_count Parameter #

When you initialize the Compare object, you can explicitly set the display_count argument to a value large enough to cover all your differences. If you want to show every single mismatch, you can even calculate it based on the size of your datasets:

import datacompy
import pandas as pd

# Load your large files (adjust read method as needed for your file type)
df1 = pd.read_csv('large_file1.csv')
df2 = pd.read_csv('large_file2.csv')

# Initialize Compare with a custom display count
max_possible_diff = max(len(df1), len(df2))
compare = datacompy.Compare(
    df1,
    df2,
    join_columns='your_key_column',  # Replace with your actual join column(s)
    display_count=max_possible_diff  # Show all differences
)

# Generate the full report
full_report = compare.report()
print(full_report)  # Or save to file (see next tip)

2. Save the Report to a File #

For extremely large datasets, printing the report to the console might still cause truncation or performance issues. Instead, write the full report directly to a text file—this way you can open it in any text editor and browse all content:

with open('complete_comparison_report.txt', 'w', encoding='utf-8') as report_file:
    report_file.write(compare.report())

3. Extract Raw Difference DataFrames #

If you don’t need the formatted report and just want to work with the raw mismatch data, Datacompy gives you direct access to dedicated DataFrames for all types of differences. You can export these to CSV (or any format) for full analysis:

# Export rows that exist in both but have mismatched values
compare.rows_mismatch.to_csv('mismatched_rows.csv', index=False)

# Export rows unique to the first dataframe
compare.df1_unq_rows.to_csv('df1_unique_rows.csv', index=False)

# Export rows unique to the second dataframe
compare.df2_unq_rows.to_csv('df2_unique_rows.csv', index=False)

These methods should let you bypass the 10-row limit and get complete visibility into all differences between your large files.

内容的提问来源于stack exchange，提问作者Momo