K-means++ Algorithm - ML

Clustering is used to group similar data points. K-Means is a commonly used clustering method, but it often gives poor results because the initial cluster centers are chosen randomly. This may lead to empty clusters, overlapping clusters or centroids falling too close to each other.

For example, the outputs below show how K-Means can form incorrect clusters due to weak initialization. To fix this, K-Means++ was introduced. It improves the way initial centroids are selected so that the clustering becomes more stable, accurate and faster.

K-Means++ is an improved version of the K-Means algorithm. Instead of picking all centroids randomly, it chooses the first center randomly and then selects the remaining centers in a spaced-out manner. This ensures:

• Better cluster separation.
• Faster convergence.
• More consistent results compared to K-Means.

Working

K-means++ revolutionizes the initialization step while keeping the rest of the K-means algorithm intact. The key insight is deceptively simple: spread out the initial centers as much as possible.

The Initialization Process

The algorithm follows these steps:

1. First center: Choose the first cluster center uniformly at random from the data points

2. Subsequent centers: For each remaining center:

• Calculate the distance from each data point to its nearest existing center.
• Choose the next center with probability proportional to the square of this distance.
• Points farther from existing centers have a higher chance of being selected.

3. Standard K-means: Once all k centers are initialized, proceed with the standard K-means algorithm

The squared distance weighting is important it ensures that points far from existing centers are much more likely to be chosen, naturally spreading the centers across the data space.

Note: Although the initialization in K-means++ is computationally more expensive than the standard K-means algorithm, the run-time for convergence to optimum is drastically reduced for K-means++. This is because the centroids that are initially chosen are likely to lie in different clusters already.

Mathematical Foundation

Let's formalize the selection probability. When choosing the (i+1)-th center, the probability of selecting point x is:

Where D(x) is the shortest distance from point x to any already-chosen center and the sum runs over all data points. his probability distribution is called D²-weighting and it's the fundamental principle that makes K-means++ work so well.

Implementation in Python

Let's understand how KMeans++ initializes centroids step by step using the following implementation:

1. Dataset Creation

Four separate Gaussian clusters are generated with different means and covariances to simulate different groupings in the data.

2. Plotting Helper Function

This function is used to visualize the data points and the selected centroids at each step. All data points are shown in gray.

• Previously selected centroids are marked in black.
• The current centroid being added is marked in red.
• This helps visualize the centroid initialization process step by step.

3. Euclidean Distance Function

This is a standard formula to compute the distance between two vectors p1 and p2 in 2D space.

4. K-Means++ Initialization

This function selects initial centroids using the K-Means++ strategy. The first centroid is chosen randomly from the dataset. For the next centroids:

• It calculates the distance of every point to its nearest existing centroid.
• Chooses the next centroid randomly, where points farther from existing centroids have a higher probability of being selected.

Output: 

It shows the dataset with the first randomly selected centroid (in red). No black points are visible since only one centroid is selected.

The second centroid is selected which is the farthest point from the first centroid. The first centroid becomes black and the new centroid is marked in red

The third centroid is selected. The two previously selected centroids are shown in black while the newly selected centroid is in red.

The final centroid is selected completing the initialization. Three previously selected centroids are in black and the last selected centroid is in red.

You can download complete code from here.

Applications

• Image segmentation: Divides images into regions based on color or texture, useful for tasks like object recognition and tracking.
• Customer segmentation: Groups customers based on behavior or demographics to improve targeted marketing and advertising.        
• Recommender systems: Suggests products or services using user preferences and past data, commonly used in e-commerce platforms.