如何通过Spark中K-means模型的Within Set Sum of Squared Errors值计算准确率？

Great question—let’s break this down clearly, since it’s a common mix-up between supervised and unsupervised learning metrics.

First: Why WSSSE (from computeCost) Doesn’t Equal Accuracy #

The computeCost method (or cost in newer Spark versions) returns the Within Set Sum of Squared Errors, which measures how tightly your data points cluster around their assigned center. A lower value means your clusters are more compact.

But here’s the critical point: accuracy is a supervised learning metric—it requires knowing "true" labels for your data to compare against model predictions. K-means is unsupervised, so without ground truth labels, "accuracy" isn’t a meaningful term here. WSSSE tells you how well the model grouped similar points, but it can’t confirm if those groups match any predefined "correct" categories.

If You Do Have True Labels (Semi-Supervised Scenario) #

If your dataset has known class labels (and you’re using K-means as a semi-supervised tool), you can calculate metrics that measure alignment between clusters and true labels, or even derive an accuracy-like score:

1. Adjusted Rand Index (ARI) or Adjusted Mutual Information (AMI) #

These metrics directly compare clustering results to true labels, accounting for random chance (adjusted versions are far more reliable than their unadjusted counterparts). Spark’s ClusteringEvaluator simplifies this:

import org.apache.spark.ml.evaluation.ClusteringEvaluator

// Assume your DataFrame has columns: "features", "trueLabel", "prediction" (cluster ID)
val evaluator = new ClusteringEvaluator()
  .setPredictionCol("prediction")
  .setLabelCol("trueLabel")
  .setMetricName("adjustedRandIndex") // Use "adjustedMutualInfo" for AMI

val alignmentScore = evaluator.evaluate(yourPredictionDF)
// Score ranges from 0 (no alignment) to 1 (perfect alignment)

2. Map Cluster IDs to True Labels, Then Calculate Accuracy #

Cluster IDs are arbitrary (e.g., cluster 0 might correspond to true label 2), so first map each cluster to the most frequent true label within it. Then you can use standard classification accuracy:

import org.apache.spark.ml.evaluation.MulticlassMetrics
import org.apache.spark.sql.functions._

// Step 1: Find the most common true label for each cluster
val clusterLabelMap = yourPredictionDF
  .groupBy("prediction", "trueLabel")
  .count()
  .orderBy(desc("count"))
  .groupBy("prediction")
  .agg(first("trueLabel").alias("mappedLabel"))
  .collect()
  .map(row => (row.getInt(0), row.getInt(1)))
  .toMap

// Step 2: Create a UDF to map cluster IDs to true labels
val mapClusterToLabel = udf((clusterId: Int) => clusterLabelMap(clusterId))

// Step 3: Calculate accuracy with MulticlassMetrics
val labeledPredictions = yourPredictionDF
  .withColumn("predictedLabel", mapClusterToLabel(col("prediction")))
  .select(col("predictedLabel").cast(DoubleType), col("trueLabel").cast(DoubleType))

val metrics = new MulticlassMetrics(labeledPredictions.rdd)
val accuracy = metrics.accuracy

If You Don’t Have True Labels #

In pure unsupervised clustering, "accuracy" isn’t applicable. Instead, use these metrics to evaluate cluster quality:

• WSSSE: Use this with the elbow method to pick the optimal number of clusters (plot WSSSE vs. K; the "elbow" point is where adding more clusters stops reducing WSSSE significantly).
• Silhouette Score: Measures how similar a point is to its own cluster vs. other clusters. Ranges from -1 (poor clustering) to 1 (excellent clustering). Compute it with ClusteringEvaluator using setMetricName("silhouette").

内容的提问来源于stack exchange，提问作者Ramkumar