Scikit-learn处理形状可变数据集的技术咨询

Hey there! Let's tackle your unsupervised clustering and visualization problem step by step—since your dataset has those variable-row, fixed-column tables per sample, the first big hurdle is turning those messy tables into consistent, usable features that clustering algorithms can work with. Here's a practical, actionable plan:

Clustering algorithms need fixed-length feature vectors, so we'll extract statistical/structural features from each sample's table. Let's break this down by column type:

Column-wise Feature Extraction #

For each column in your tables (like column1, column2), pull relevant stats based on what kind of data it holds:

• Numeric columns: Mean, median, standard deviation, min/max, non-null count, skewness, kurtosis
• Datetime columns (like your column2 timestamps): Min/max timestamp, time range (in hours/minutes), count of entries, even hourly distribution if that makes sense
• Categorical/symbol columns (like column1 with "-"): Top category frequency, ratio of the most common value, number of unique values

Here's a quick Python snippet to implement this (using pandas, since it's perfect for table manipulation):

import pandas as pd
import numpy as np

def extract_table_features(table_dict):
    # Convert the table dict to a DataFrame
    df = pd.DataFrame(table_dict)
    features = {}
    
    for col in df.columns:
        col_data = df[col]
        
        # Handle datetime columns
        if pd.api.types.is_datetime64_any_dtype(col_data):
            features[f"{col}_min_ts"] = col_data.min().timestamp()
            features[f"{col}_max_ts"] = col_data.max().timestamp()
            features[f"{col}_hour_range"] = (col_data.max() - col_data.min()).total_seconds() / 3600
            features[f"{col}_entry_count"] = col_data.count()
        
        # Handle numeric columns
        elif pd.api.types.is_numeric_dtype(col_data):
            features[f"{col}_mean"] = col_data.mean()
            features[f"{col}_median"] = col_data.median()
            features[f"{col}_std"] = col_data.std()
            features[f"{col}_non_nulls"] = col_data.count()
        
        # Handle categorical/symbol columns
        else:
            val_counts = col_data.value_counts(normalize=True)
            if len(val_counts) > 0:
                features[f"{col}_top_val"] = pd.factorize([val_counts.index[0]])[0][0]  # Encode text to num
                features[f"{col}_top_val_ratio"] = val_counts.iloc[0]
            else:
                features[f"{col}_top_val"] = 0
                features[f"{col}_top_val_ratio"] = 0
            features[f"{col}_unique_count"] = col_data.nunique()
    
    # Return as a numpy array for clustering
    return np.array(list(features.values()))

# Process all your samples
all_features = []
for sample in your_dataset:
    table_feats = extract_table_features(sample["table1"])
    all_features.append(table_feats)
all_features = np.array(all_features)

Standardize Your Features #

Different features will have wildly different scales (e.g., timestamps are huge numbers, while ratios are 0-1). Normalize them so no single feature dominates the clustering:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
scaled_features = scaler.fit_transform(all_features)

With 1000 samples, these algorithms work great—pick based on what you need:

K-Means (Great for Quick Exploration) #

K-Means is fast and easy to interpret, but you need to pick the right number of clusters. Use the silhouette score to find the best k:

from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
import matplotlib.pyplot as plt

# Test k from 2 to 9
sil_scores = []
k_range = range(2, 10)
for k in k_range:
    kmeans = KMeans(n_clusters=k, random_state=42)
    labels = kmeans.fit_predict(scaled_features)
    sil_scores.append(silhouette_score(scaled_features, labels))

# Pick the k with the highest silhouette score
best_k = k_range[sil_scores.index(max(sil_scores))]
kmeans = KMeans(n_clusters=best_k, random_state=42)
cluster_labels = kmeans.fit_predict(scaled_features)

DBSCAN (Great for Odd-Shaped Clusters & Outliers) #

DBSCAN doesn't require you to set the number of clusters upfront, and it automatically flags outliers (marked as -1):

from sklearn.cluster import DBSCAN

# Tune eps (neighborhood radius) and min_samples (min points in a neighborhood)
dbscan = DBSCAN(eps=0.5, min_samples=5)
cluster_labels = dbscan.fit_predict(scaled_features)

Hierarchical Clustering (Great for Understanding Relationships) #

If you want to see how samples group into nested clusters, use hierarchical clustering and plot a dendrogram:

from scipy.cluster.hierarchy import dendrogram, linkage

linked = linkage(scaled_features, method='ward')  # Ward minimizes variance between clusters
plt.figure(figsize=(15, 8))
dendrogram(linked, orientation='top', distance_sort='descending', show_leaf_counts=True)
plt.title("Hierarchical Clustering Dendrogram")
plt.show()

Since your features are high-dimensional, you'll need to reduce them to 2D first. These two methods work best:

PCA (Linear Dimensionality Reduction) #

PCA is fast and preserves global structure:

from sklearn.decomposition import PCA

# Reduce to 2 dimensions
pca = PCA(n_components=2)
pca_feats = pca.fit_transform(scaled_features)

# Plot clusters
plt.figure(figsize=(10, 6))
unique_labels = np.unique(cluster_labels)
for label in unique_labels:
    mask = cluster_labels == label
    plt.scatter(pca_feats[mask, 0], pca_feats[mask, 1], label=f"Cluster {label}", alpha=0.6)
plt.legend()
plt.title("Clusters (PCA 2D Projection)")
plt.xlabel("PCA Component 1")
plt.ylabel("PCA Component 2")
plt.show()

t-SNE (Non-Linear Dimensionality Reduction) #

t-SNE is better at capturing local, complex cluster structures—perfect if your clusters are non-linear:

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, random_state=42, perplexity=30)  # Perplexity ~ sqrt(n_samples)
tsne_feats = tsne.fit_transform(scaled_features)

plt.figure(figsize=(10, 6))
for label in unique_labels:
    mask = cluster_labels == label
    plt.scatter(tsne_feats[mask, 0], tsne_feats[mask, 1], label=f"Cluster {label}", alpha=0.6)
plt.legend()
plt.title("Clusters (t-SNE 2D Projection)")
plt.show()

• If your tables have sequential data (like time series), consider using sequence embeddings (e.g., a tiny LSTM) or Fourier transforms to capture temporal patterns
• After clustering, dig into the feature stats for each cluster—compute mean values for each feature across clusters to see what makes them distinct
• Don't forget to handle missing values first! For numeric columns, fill with median/mean; for categorical, fill with the most frequent value or a "missing" category

内容的提问来源于stack exchange，提问作者denisb411