如何在DataFrame中新增列存储每行[A1,A2]与[B1,B2]的相关系数

Instead of splitting your DataFrame, you can use vectorized operations (the fastest approach for pandas) to calculate the Pearson correlation coefficient between [A1,A2] and [B1,B2] for each row. Here's how to do it step by step:

Step-by-Step Vectorized Solution #

First, let's set up your sample DataFrame:

import pandas as pd

data = {
    'A1': [1,3,6,2,5],
    'A2': [5,3,4,7,9],
    'B1': [7,4,5,6,1],
    'B2': [6,5,7,8,3]
}
df = pd.DataFrame(data)

Now compute the correlation coefficients using vectorized calculations (no loops or row-wise apply):

# Calculate row-wise means for A and B columns
mean_a = df[['A1', 'A2']].mean(axis=1)
mean_b = df[['B1', 'B2']].mean(axis=1)

# Compute numerator of the correlation formula
numerator = (df['A1'] - mean_a) * (df['B1'] - mean_b) + (df['A2'] - mean_a) * (df['B2'] - mean_b)

# Compute sum of squared deviations for A and B
sum_sq_a = (df['A1'] - mean_a)**2 + (df['A2'] - mean_a)**2
sum_sq_b = (df['B1'] - mean_b)**2 + (df['B2'] - mean_b)**2

# Calculate correlation coefficient (handles division by zero with NaN)
df['Corr_Coeff'] = numerator / (sum_sq_a**0.5 * sum_sq_b**0.5)

Result: #

A1  A2  B1  B2  Corr_Coeff
0   1   5   7   6        -1.0
1   3   3   4   5         NaN
2   6   4   5   7        -1.0
3   2   7   6   8         1.0
4   5   9   1   3         1.0

Alternative: Using apply with numpy.corrcoef #

If readability is more important than raw speed (for small datasets), you can use df.apply with numpy.corrcoef:

import numpy as np

df['Corr_Coeff'] = df.apply(
    lambda row: np.corrcoef([row['A1'], row['A2']], [row['B1'], row['B2']])[0, 1],
    axis=1
)

This gives the same result, but vectorized operations are significantly faster for large DataFrames (since apply loops through each row individually).

Key Notes: #

• When either [A1,A2] or [B1,B2] are constant values (e.g., row 1 where A1=A2=3), the correlation coefficient is undefined (result is NaN), which is statistically correct.
• The vectorized approach leverages pandas' optimized backend, making it ideal for large datasets with thousands/millions of rows.

内容的提问来源于stack exchange，提问作者Gautham Kanthasamy