如何对pandas的df_jpas下采样，使其z分布匹配df_gaia的z分布？

核心思路 #

通过密度匹配的方式，让df_jpas在目标红移区间（0.8<z<2.3）的样本分布与df_gaia对齐——本质是给df_jpas的样本分配对应权重，再按权重进行随机下采样。

具体步骤 #

1. 筛选目标区间样本 #

先将两个数据表限定在目标红移范围内：

import pandas as pd
import numpy as np

# 筛选0.8<z<2.3的样本
df_jpas_target = df_jpas[(df_jpas['z'] > 0.8) & (df_jpas['z'] < 2.3)].copy()
df_gaia_target = df_gaia[(df_gaia['z'] > 0.8) & (df_gaia['z'] < 2.3)].copy()

2. 计算红移分布密度 #

用核密度估计（KDE）分别计算两个样本的红移概率密度：

from scipy.stats import gaussian_kde

# 生成KDE模型
kde_gaia = gaussian_kde(df_gaia_target['z'], bw_method='scott')
kde_jpas = gaussian_kde(df_jpas_target['z'], bw_method='scott')

3. 计算样本匹配权重 #

对df_jpas的每个样本，根据其红移对应的密度比值计算权重（权重=Gaia密度/JPAS密度），同时处理极端值：

# 获取每个JPAS样本红移对应的密度值
jpas_z_density = kde_jpas(df_jpas_target['z'])
gaia_z_density = kde_gaia(df_jpas_target['z'])

# 计算权重，避免除以0或极端值
weights = gaia_z_density / jpas_z_density
weights[np.isinf(weights)] = 0
weights[np.isnan(weights)] = 0
# 限制权重上限，防止少数样本被过度选择
weights = np.clip(weights, 0, np.percentile(weights, 99))

4. 按权重随机下采样 #

根据权重对df_jpas_target抽样，样本量可匹配df_gaia_target的规模：

# 设置采样数量（与Gaia目标区间样本量一致）
sample_size = len(df_gaia_target)

# 按权重无放回抽样
df_jpas_downsampled = df_jpas_target.sample(
    n=sample_size, 
    weights=weights, 
    replace=False, 
    random_state=42
)

5. 验证匹配效果 #

绘制下采样后的分布与Gaia分布对比：

import matplotlib.pyplot as plt

plt.hist(df_gaia_target['z'], bins=50, alpha=0.5, label='Gaia')
plt.hist(df_jpas_downsampled['z'], bins=50, alpha=0.5, label='JPAS Downsampled')
plt.xlabel('Redshift z')
plt.ylabel('Number of Galaxies')
plt.legend()
plt.show()

替代方案：分箱匹配 #

如果KDE计算效率低，可采用分箱匹配法：

• 将0.8-2.3区间分成若干bins
• 按每个bin内Gaia与JPAS的样本数量比值，对JPAS逐bin下采样

# 生成红移分箱
bins = np.linspace(0.8, 2.3, 30)
df_jpas_target['bin'] = np.digitize(df_jpas_target['z'], bins)
df_gaia_target['bin'] = np.digitize(df_gaia_target['z'], bins)

# 计算每个bin的采样比例
bin_counts_gaia = df_gaia_target['bin'].value_counts().sort_index()
bin_counts_jpas = df_jpas_target['bin'].value_counts().sort_index()
sample_ratios = bin_counts_gaia / bin_counts_jpas
sample_ratios = sample_ratios.fillna(0)

# 逐bin采样
downsampled_list = []
for bin_idx, ratio in sample_ratios.items():
    bin_data = df_jpas_target[df_jpas_target['bin'] == bin_idx]
    if ratio > 0:
        sample_num = max(1, int(len(bin_data) * ratio))
        downsampled_bin = bin_data.sample(n=sample_num, random_state=42)
        downsampled_list.append(downsampled_bin)

df_jpas_downsampled = pd.concat(downsampled_list).drop('bin', axis=1)

内容的提问来源于stack exchange，提问作者NeStack