Implementing Agglomerative Clustering using Sklearn

Agglomerative Clustering is one of the most common hierarchical clustering technique where each data point starts in its own group (cluster) and step by step the closest clusters are joined together until we reach one big cluster. It's a bottom-up approach meaning:

• Each data point starts in its own cluster.
• Pairs of clusters are merged step-by-step based on a linkage criterion like shortest distance until all points are merged into a single cluster or until a desired number of clusters is formed.

It's used in marketing, biology, education and fraud detection to find natural patterns in data. Let's implement it step by step:

Step 1: Importing the required libraries 

First we will import all the necessary libraries like numpy ,pandas, matplotlib and scikit learn.

Step 2: Loading and Cleaning the data 

We will now read the .csv file and clean it.

• Remove the CUST_ID column since it's just an ID and not useful
• Handle missing values using forward fill.

You can download the dataset from here

Output:

Step 3: Preprocessing the data 

We prepare the data so that all features are on the same scale.

• Scalingmakes features comparable It is important because clustering depends on distance.
• Normalizationhelps the clustering algorithm work better.

Step 4: Reducing the dimensionality of the Data 

We use PCA to reduce many columns features to just 2 so we can easily visualize the data.

Step 5: Make the Dendrograms 

A dendrogramhelps us decide how many clusters to choose. We will use the matplotlib to plot it.

👁 Image

To determine the optimal number of clusters by visualizing the data, imagine all the horizontal lines as being completely horizontal and then after calculating the maximum distance between any two horizontal lines, draw a horizontal line in the maximum distance calculated.

👁 Image
The above image shows that the optimal number of clusters should be 2 for the given data.

Step 6: Apply Agglomerative Clustering for Different Values of k

Now let’s apply clustering for different values of k (number of clusters). For each value of k we created a clustering model and plot the two PCA components colored by cluster.

Output:

Step 7: Evaluate models and Visualizing results

Silhouette score tells us how well the data has been grouped. The Higher the score the better is model.

Output:

As in the above image based on the Silhouette Score and Dendrogram we usually choose the value of k that gives the highest score. In most cases with this dataset the best number of clusters is 2.

To download the complete code: click here