高不平衡医学数据（含4k二元/100数值变量）聚类优化方案咨询

Hey there! Let’s break down how to tackle this tricky clustering problem with your imbalanced, mixed-type medical dataset—your initial SVD+DBSCAN approach likely struggled because it doesn’t play well with either the mixed binary/numeric features or the uneven patient subgroup distributions. Here are targeted alternatives to try:

First, let’s address the high dimensionality and mixed feature types, which are probably throwing off your downstream clustering:

• UMAP with Gower Distance: Unlike linear methods like SVD, UMAP excels at preserving local data structure (critical for finding meaningful patient subgroups). The key here is using the Gower distance—it’s designed explicitly for mixed-type data, assigning appropriate weights to binary and numeric features. Most UMAP implementations (like umap-learn) let you specify metric='gower' directly, so you don’t have to force-fit your data into a linear space that doesn’t suit it.
• Prior Feature Selection: 4000 binary features is a lot—many are likely redundant (e.g., rare symptoms with almost no variation). Start by pruning these:
  • Use variance thresholding to drop binary features that are 99%+ 0 or 1.
  • Try unsupervised feature selection methods like Clustering Feature Selection (CFS) to keep only features that add unique clustering signal.
  • For numeric features, use correlation analysis to remove highly correlated ones before combining with binary data.
• t-SNE (with Caveats): If you’re focused on visualizing local subgroups first, t-SNE works well—but it’s slower for large datasets. Pair it with Gower distance too, and tweak the perplexity parameter based on your sample size (aim for 5-50, depending on how many patients you have).

Once your features are prepped, switch to algorithms that handle imbalance and mixed data better than DBSCAN:

• HDBSCAN: This is DBSCAN’s smarter cousin. It automatically detects clusters of varying densities, which is perfect for imbalanced datasets where some patient subgroups are small and sparse. You can run it directly on your Gower-distance matrix (or UMAP-reduced data) without manually tuning the eps parameter—HDBSCAN finds the optimal density thresholds on its own.
• Gower-Based K-Means or Hierarchical Clustering:
  • Gower K-Means: Standard K-Means assumes Euclidean distance, which doesn’t make sense for binary features. Instead, use a K-Means variant that uses Gower distance. Libraries like scikit-learn let you customize the distance metric, or you can use specialized implementations for mixed data.
  • Hierarchical Clustering with Gower Distance: This method builds a tree of clusters (dendrogram), which is great for medical data because it lets you interpret how subgroups relate to each other. Use average linkage or Ward’s method to balance between preserving small clusters and merging similar ones.
• Latent Class Analysis (LCA): LCA is a model-based clustering technique designed specifically for mixed-type data. It assumes each patient subgroup is a "latent class" with distinct distributions for binary (Bernoulli) and numeric (Gaussian) features. It’s widely used in medical research because it produces interpretable results—you can even get probabilities for each patient belonging to a class.
• Weighted Gaussian Mixture Models (GMM): If you prefer model-based methods, tweak GMM to handle mixed data: use Bernoulli distributions for binary features and Gaussian for numeric ones. You can also weight samples to prioritize smaller, underrepresented subgroups, helping the model not just focus on the largest cluster.

• Consensus Clustering: For more reliable results (critical in medical settings), combine outputs from multiple algorithms (e.g., HDBSCAN + LCA + hierarchical clustering) using consensus clustering. This reduces the chance of getting a fluke result from a single method.
• Domain Knowledge Checks: Always validate clusters against clinical expertise. For example, if a cluster is supposed to represent patients with a specific condition, cross-check if their features align with known symptoms. You can even weight clinically relevant features higher during preprocessing to guide the algorithm.
• SMOTE-NC for Imbalance: If your dataset has extreme imbalance (e.g., one subgroup is 5% of the total), try SMOTE-NC—a variant of SMOTE designed for mixed-type data. It generates synthetic samples for small subgroups to help clustering algorithms pick up on their patterns without overwhelming the data.

内容的提问来源于stack exchange，提问作者Rushabh Patel