scikit-learn层次聚类可视化及相关技术问题咨询

背景与现有实现 #

我有一个形状为(4312, 1024)的嵌入矩阵，对应4312个英文句子的1024维嵌入向量，希望通过聚类并可视化结果，验证距离阈值的合理性。

聚类代码 #

import numpy as np
from sklearn.cluster import AgglomerativeClustering

model = AgglomerativeClustering(n_clusters=None, metric='cosine', 
                                compute_full_tree='auto', 
                                linkage='complete', 
                                distance_threshold=0.2,
                                compute_distances=True)
clustering = model.fit(embeddings)
print(f'Number of clusters: {clustering.n_clusters_}')
print(f'Labels:\n{clustering.labels_}')
# count unique labels
unique_labels, counts = np.unique(clustering.labels_, return_counts=True)
print(f'Number of clusters by counting: {len(unique_labels)}')

# Sort in descending order of counts
sorted_indices = np.argsort(-counts)
unique_labels = unique_labels[sorted_indices]
counts = counts[sorted_indices]
print(f'Unique labels: {unique_labels}')
print(f'counts: {counts}')

聚类结果 #

Number of clusters: 1714
clustering.labels_:
[ 460  820  245 ... 1030  112 1367]
Number of clusters by counting: 1714
Unique labels: [ 410  352  229 ... 1039 1041 1713]
counts: [55 42 33 ...  1  1  1]

得到1714个聚类，最大聚类包含55个样本；将距离阈值提高到0.25时，聚类数量减少至1395。为了观察阈值从0.2提升到0.25时的合并情况，绘制了阈值为0.2时的树状图，代码如下：

树状图绘制代码 #

from matplotlib import pyplot as plt
from scipy.cluster.hierarchy import dendrogram

def plot_dendrogram(model, **kwargs):
    # Create linkage matrix and then plot the dendrogram

    # create the counts of samples under each node
    counts = np.zeros(model.children_.shape[0])
    n_samples = len(model.labels_)
    for i, merge in enumerate(model.children_):
        current_count = 0
        for child_idx in merge:
            if child_idx < n_samples:
                current_count += 1  # leaf node
            else:
                current_count += counts[child_idx - n_samples]
        counts[i] = current_count

    linkage_matrix = np.column_stack(
        [model.children_, model.distances_, counts]
    ).astype(float)

    # Plot the corresponding dendrogram
    dendrogram(linkage_matrix, **kwargs)

plt.title("Hierarchical Clustering Dendrogram")
# plot the top three levels of the dendrogram
plot_dendrogram(clustering, truncate_mode="level", p=3, distance_sort='ascending', show_leaf_counts=True)
plt.xlabel("Number of points in node (or index of point if no parenthesis).")
plt.show()

绘制的树状图展示了层次聚类的顶部三层结构。

问题 #

1. 我以为cosine距离的取值范围是[0,1]，但树状图中叶节点到父节点的距离却大于1，这是数值误差导致的吗？
2. 当前树状图是从原始样本开始自上而下展示的，能否改为以现有的1714个聚类为叶节点（标注每个聚类的样本数量，如(55)、(42)），自下而上展示这些聚类如何根据距离合并？要求叶节点恰好1714个，逐层向上合并直到指定层数，形成不连通组件。

解答 #

问题1：关于cosine距离大于1的原因 #

这不是数值误差，而是对cosine距离的定义理解有误：
Scikit-learn中metric='cosine'计算的是1 - cosine相似度，而cosine相似度的取值范围是[-1, 1]（嵌入向量通常包含正负分量），因此cosine距离的取值范围是**[0, 2]**，大于1的距离是完全合理的。只有当所有向量都是非负的（如TF-IDF向量），cosine相似度才会落在[0,1]区间，对应的距离才是[0,1]。

问题2：以聚类为叶节点的树状图实现 #

要实现需求，需要以现有聚类的中心向量为基础，重新执行层次聚类，再自定义树状图的叶节点标注。具体步骤如下：

步骤1：计算现有聚类的中心与样本数量

# 按排序后的聚类标签，收集每个聚类的样本并计算中心
cluster_samples = {}
for idx, label in enumerate(clustering.labels_):
    if label not in cluster_samples:
        cluster_samples[label] = []
    cluster_samples[label].append(idx)

cluster_centers = []
sorted_cluster_sizes = []
# 用之前排序好的unique_labels遍历，保证顺序与counts一致
for label in unique_labels:
    samples = embeddings[cluster_samples[label]]
    cluster_centers.append(np.mean(samples, axis=0))
    sorted_cluster_sizes.append(len(samples))
cluster_centers = np.array(cluster_centers)

步骤2：对聚类中心执行层次聚类（模拟阈值从0.2到0.25的合并）

cluster_model = AgglomerativeClustering(
    n_clusters=None, metric='cosine', linkage='complete',
    distance_threshold=0.05,  # 对应阈值从0.2提升到0.25的差值
    compute_distances=True
)
cluster_clustering = cluster_model.fit(cluster_centers)

步骤3：绘制自定义树状图（标注聚类样本数量）

def plot_cluster_dendrogram(model, cluster_sizes, **kwargs):
    counts = np.zeros(model.children_.shape[0])
    n_clusters = len(cluster_sizes)
    for i, merge in enumerate(model.children_):
        current_count = 0
        for child_idx in merge:
            if child_idx < n_clusters:
                current_count += cluster_sizes[child_idx]
            else:
                current_count += counts[child_idx - n_clusters]
        counts[i] = current_count

    linkage_matrix = np.column_stack(
        [model.children_, model.distances_, counts]
    ).astype(float)

    # 自定义叶节点标签为聚类的样本数量
    dendrogram(linkage_matrix, leaf_label_func=lambda x: f"({cluster_sizes[x]})", **kwargs)

plt.title("Cluster-level Hierarchical Clustering Dendrogram")
# 绘制顶部3层，按距离升序排列
plot_cluster_dendrogram(cluster_clustering, sorted_cluster_sizes, 
                        truncate_mode="level", p=3, 
                        distance_sort='ascending', show_leaf_counts=True)
plt.xlabel("Number of samples in cluster (or merged cluster size)")
plt.show()

这样生成的树状图会以1714个聚类为叶节点，标注对应的样本数量，逐层展示它们在阈值提升到0.25时的合并过程，符合需求。

内容的提问来源于stack exchange，提问作者f10w