基于Jensen-Shannon散度的分半信度：矩阵构建与计算问题

步骤1：合并数据并设置复合索引 #

首先将两个DataFrame合并，用ID和group作为复合索引，确保矩阵的行和列能唯一标识每个个体：

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

# 原始数据
df_1 = pd.DataFrame({'ID': ['a', 'b', 'c', 'd', 'e', 'f'], 
                     'response': [["apple", "berry", "cherry"],
                                  ["pear", "pineapple", "plum"],
                                  ["blue_berry"],
                                  ["orange", "lemon"],
                                  ["tomato", "pumpkin"],
                                  ["avocado", "strawberry"]], 
                     'group': [1, 2, 1, 2, 1, 2]})

df_2 = pd.DataFrame({'ID': ['A', 'B','C', 'D', 'E', 'F'], 
                     'response': [["pear", "plum", "cherry"],
                                  ["orange", "lemon", "lime", "pineapple"],
                                  ["pumpkin"],
                                  ["tomato", "strawberry"],
                                  ["avocado", "apple"],
                                  ["berry", "cherry", "apple"]], 
                     'group': [1, 2, 1, 2, 1, 2]})

# 合并DataFrame并设置复合索引
df_combined = pd.concat([df_1, df_2], ignore_index=True)
df_combined = df_combined.set_index(['ID', 'group'])

步骤2：转换响应为统一概率分布 #

JSD要求输入是等长且和为1的概率分布，因此先收集所有出现过的响应项，再将每个个体的响应列表转换为对应分布：

# 收集所有唯一响应项
all_items = set()
for resp in df_combined['response']:
    all_items.update(resp)
all_items = sorted(list(all_items))

# 将响应列表转为概率分布
def response_to_dist(response):
    dist = np.zeros(len(all_items))
    for item in response:
        idx = all_items.index(item)
        dist[idx] += 1
    total = dist.sum()
    return dist / total if total != 0 else dist

# 生成所有个体的分布矩阵
dist_matrix = np.array([response_to_dist(resp) for resp in df_combined['response']])

步骤3：修正JSD函数处理零值 #

原函数未处理零值会导致NaN，加入小epsilon避免除以0和对数无意义的情况：

def jsdiv(P, Q, epsilon=1e-10):
    """计算两个概率分布的Jensen-Shannon散度"""
    def _kldiv(A, B):
        # 加入epsilon避免数值异常
        A = A + epsilon
        B = B + epsilon
        A = A / A.sum()
        B = B / B.sum()
        return np.sum(A * np.log2(A / B))
    
    P = np.array(P)
    Q = np.array(Q)
    M = 0.5 * (P + Q)
    return 0.5 * (_kldiv(P, M) + _kldiv(Q, M))

步骤4：构建JSD得分矩阵 #

遍历所有个体对，计算两两JSD得分，生成方阵：

n = len(df_combined)
jsd_matrix = np.zeros((n, n))

for i in range(n):
    for j in range(n):
        jsd_matrix[i, j] = jsdiv(dist_matrix[i], dist_matrix[j])

# 转换为带复合索引的DataFrame
jsd_df = pd.DataFrame(jsd_matrix, index=df_combined.index, columns=df_combined.index)

步骤5：绘制热力图 #

用seaborn可视化JSD矩阵，直观展示个体响应的相似度（JSD越小，分布越相似）：

plt.figure(figsize=(12, 10))
sns.heatmap(jsd_df, annot=True, cmap='coolwarm', fmt='.4f')
plt.title('个体响应的Jensen-Shannon散度热力图')
plt.xlabel('个体（ID+Group）')
plt.ylabel('个体（ID+Group）')
plt.show()

内容的提问来源于stack exchange，提问作者psychcoder