Apache Spark与scikit-learn的KMeans聚类WSSE结果不一致问题

Great question—this is a common pitfall when comparing distributed KMeans implementations to their single-machine counterparts. Let’s break down why your WSSE increased when moving from k=5 to k=6 in PySpark, and why adjusting initSteps resolved the issue:

1. Core Differences in Initialization: Spark vs. scikit-learn #

The root cause lies in how each library handles initial centroid selection:

• scikit-learn's KMeans: Defaults to k-means++ initialization with n_init=10—it runs 10 separate initializations and picks the result with the lowest WSSE. This ensures it avoids poor local optima most of the time.
• PySpark's KMeans: Uses k-means|| (a distributed variant of k-means++), but defaults to only initSteps=2. This means the algorithm samples very few candidate centroids during initialization, making it far more likely to land on a suboptimal starting point.

When you increased k from 5 to 6, Spark’s limited initialization steps just happened to generate a set of centroids that led to a worse (higher WSSE) clustering outcome than k=5. Scikit-learn’s multi-round initialization prevented this by always selecting the best possible starting point.

2. Why Adjusting tol or maxIter Didn’t Help #

Parameters like tol (convergence threshold) and maxIter (maximum iterations) control how the model converges once initialized—they don’t fix poor initial centroids. If your starting points are bad, the model will just converge to the nearest local optimum, which can still have a higher WSSE than a lower k-value run with better initialization. Tweaking these parameters only makes the model converge more thoroughly to that suboptimal state, not escape it.

3. Why Increasing initSteps Fixed the Problem #

The initSteps parameter controls how many sampling rounds the k-means|| algorithm uses to generate initial centroids. More steps mean:

• A larger pool of candidate centroids
• Centroids that are more spread out across the data
• A much higher chance of starting with centroids close to the global optimal distribution

By cranking initSteps up 50x, you gave Spark’s KMeans a far better starting point, so when it converged, the WSSE followed the expected monotonic decrease as k increased.

Bonus: Alternative Fix Using n_init #

If you don’t want to drastically increase initSteps, you can also set n_init (default 1) to a higher value (like 10, matching scikit-learn). This tells Spark to run multiple initialization rounds and pick the result with the lowest WSSE—this should also resolve the non-monotonic WSSE issue, as it adds the same "best of multiple runs" safety net that scikit-learn uses.

Recommended Solutions #

• Option 1: Increase initSteps to 50-100 to boost initialization quality
• Option 2: Set n_init=10 to run multiple initialization rounds
• Option 3: Combine both for maximum stability (especially on large datasets)

内容的提问来源于stack exchange，提问作者Clock Slave