Python多列表重叠统计存储与可视化方案咨询

1. Optimal Data Structure for Overlap Counts #

First, precompute sets for each list—set intersections are far faster than list intersections, which is critical for handling 130 keys efficiently. Below are two viable storage options:

Option 1: Pairwise Dictionary #

• Pros: Saves memory by storing only unique pairs (since overlap(A,B) equals overlap(B,A))
• Cons: Less intuitive to use for visualization

Option 2: Square Matrix (Pandas DataFrame) #

• Pros: Directly compatible with visualization tools, easy to access any pair's overlap count
• Cons: Uses O(n²) memory, but for 130 keys this equals 16,900 entries—negligible for modern systems

Code to Generate Overlap Matrix:

import pandas as pd

# Sample input data
data = {
    'key1': [10,10,11,12,15,16,18,19],
    'key2': [10,11,13,15,16,19,20],
    'key3': [10,11,11,12,15,19,21,23],
    'key4': [],
    'key5': [0],
    'key6': [10,55,66,77]
}

# Convert lists to sets for fast intersection operations
set_data = {k: set(v) for k, v in data.items()}
keys = list(set_data.keys())

# Initialize empty overlap matrix
overlap_matrix = pd.DataFrame(index=keys, columns=keys)

# Populate matrix with intersection lengths
for i in keys:
    for j in keys:
        if i == j:
            overlap_matrix.loc[i, j] = 'X'
        else:
            overlap_matrix.loc[i, j] = len(set_data[i].intersection(set_data[j]))

print(overlap_matrix)

2. Visualization Solutions #

Option 1: Styled Comparison Table #

Use pandas' Styler to create a readable, colored table where darker shades indicate higher overlap:

# Style the matrix for readability
styled_table = overlap_matrix.style \
    .set_caption("List Overlap Comparison Table") \
    .background_gradient(cmap='Blues', axis=None, subset=pd.IndexSlice[keys, keys]) \
    .set_properties(**{'text-align': 'center', 'padding': '8px'}) \
    .set_table_styles([{'selector': 'th', 'props': [('background-color', '#f0f0f0')]}])

# Save as HTML or display in Jupyter
styled_table.to_html('overlap_table.html')
# In Jupyter: display(styled_table)

Option 2: UpSet Plot #

UpSetPlot excels at showing both pairwise and higher-order intersections. Here’s how to implement it:

from upsetplot import plot, from_memberships
import matplotlib.pyplot as plt

# Map each element to the list of keys it appears in
element_to_keys = {}
for key, elements in data.items():
    for elem in elements:
        element_to_keys.setdefault(elem, []).append(key)

# Convert to membership format required by UpSetPlot
memberships = list(element_to_keys.values())
upset_data = from_memberships(memberships)

# Generate plot with count labels
plot(upset_data, show_counts=True, sort_by='cardinality')
plt.title("List Overlap UpSet Plot")
plt.tight_layout()
plt.show()

This plot displays the size of every unique intersection (including single-key sets), making it easy to spot dominant overlap patterns.

Performance Notes #

• For 130 keys, the matrix calculation runs in ~0.01 seconds (16,900 fast set operations)
• UpSetPlot handles 130 keys well, but you may want to filter out empty lists or adjust layout for readability

内容的提问来源于stack exchange，提问作者Dianna Li