Let’s say we have the below points and we want to cluster them into groups: We can assign each of these points to a separate cluster: Now, based on the similarity of these clusters, we can combine the most similar clusters together and repeat this process until only a single cluster is left: We are essentially building a hierarchy of clusters.
That’s why this algorithm is called hierarchical clustering.
I will discuss how to decide the number of clusters in a later section.
For now, let’s look at the different types of hierarchical clustering.
Types of Hierarchical Clustering There are mainly two types of hierarchical clustering: Agglomerative hierarchical clustering Divisive Hierarchical clustering Let’s understand each type in detail.
Agglomerative Hierarchical Clustering We assign each point to an individual cluster in this technique.
Suppose there are 4 data points.
We will assign each of these points to a cluster and hence will have 4 clusters in the beginning: Then, at each iteration, we merge the closest pair of clusters and repeat this step until only a single cluster is left: We are merging (or adding) the clusters at each step, right?.Hence, this type of clustering is also known as additive hierarchical clustering.
Divisive Hierarchical Clustering Divisive hierarchical clustering works in the opposite way.
Instead of starting with n clusters (in case of n observations), we start with a single cluster and assign all the points to that cluster.
So, it doesn’t matter if we have 10 or 1000 data points.
All these points will belong to the same cluster at the beginning: Now, at each iteration, we split the farthest point in the cluster and repeat this process until each cluster only contains a single point: We are splitting (or dividing) the clusters at each step, hence the name divisive hierarchical clustering.
Agglomerative Clustering is widely used in the industry and that will be the focus in this article.
Divisive hierarchical clustering will be a piece of cake once we have a handle on the agglomerative type.
Steps to Perform Hierarchical Clustering We merge the most similar points or clusters in hierarchical clustering – we know this.
Now the question is – how do we decide which points are similar and which are not?.It’s one of the most important questions in clustering!.Here’s one way to calculate similarity – Take the distance between the centroids of these clusters.
The points having the least distance are referred to as similar points and we can merge them.
We can refer to this as a distance-based algorithm as well (since we are calculating the distances between the clusters).
In hierarchical clustering, we have a concept called a proximity matrix.
This stores the distances between each point.
Let’s take an example to understand this matrix as well as the steps to perform hierarchical clustering.
Setting up the Example Suppose a teacher wants to divide her students into different groups.
She has the marks scored by each student in an assignment and based on these marks, she wants to segment them into groups.
There’s no fixed target here as to how many groups to have.
Since the teacher does not know what type of students should be assigned to which group, it cannot be solved as a supervised learning problem.
So, we will try to apply hierarchical clustering here and segment the students into different groups.
Let’s take a sample of 5 students: Creating a Proximity Matrix First, we will create a proximity matrix which will tell us the distance between each of these points.
Since we are calculating the distance of each point from each of the other points, we will get a square matrix of shape n X n (where n is the number of observations).
Let’s make the 5 x 5 proximity matrix for our example: The diagonal elements of this matrix will always be 0 as the distance of a point with itself is always 0.
We will use the Euclidean distance formula to calculate the rest of the distances.
So, let’s say we want to calculate the distance between point 1 and 2: √(10-7)^2 = √9 = 3 Similarly, we can calculate all the distances and fill the proximity matrix.
Steps to Perform Hierarchical Clustering Step 1: First, we assign all the points to an individual cluster: Different colors here represent different clusters.
You can see that we have 5 different clusters for the 5 points in our data.
Step 2: Next, we will look at the smallest distance in the proximity matrix and merge the points with the smallest distance.
We then update the proximity matrix: Here, the smallest distance is 3 and hence we will merge point 1 and 2: Let’s look at the updated clusters and accordingly update the proximity matrix: Here, we have taken the maximum of the two marks (7, 10) to replace the marks for this cluster.
Instead of the maximum, we can also take the minimum value or the average values as well.
Now, we will again calculate the proximity matrix for these clusters: Step 3: We will repeat step 2 until only a single cluster is left.
So, we will first look at the minimum distance in the proximity matrix and then merge the closest pair of clusters.
We will get the merged clusters as shown below after repeating these steps: We started with 5 clusters and finally have a single cluster.
This is how agglomerative hierarchical clustering works.
But the burning question still remains – how do we decide the number of clusters?.Let’s understand that in the next section.
How should we Choose the Number of Clusters in Hierarchical Clustering?.Ready to finally answer this question that’s been hanging around since we started learning?.To get the number of clusters for hierarchical clustering, we make use of an awesome concept called a Dendrogram.
A dendrogram is a tree-like diagram that records the sequences of merges or splits.
Let’s get back to our teacher-student example.
Whenever we merge two clusters, a dendrogram will record the distance between these clusters and represent it in graph form.
Let’s see how a dendrogram looks like: We have the samples of the dataset on the x-axis and the distance on the y-axis.
Whenever two clusters are merged, we will join them in this dendrogram and the height of the join will be the distance between these points.
Let’s build the dendrogram for our example: Take a moment to process the above image.
We started by merging sample 1 and 2 and the distance between these two samples was 3 (refer to the first proximity matrix in the previous section).
Let’s plot this in the dendrogram: Here, we can see that we have merged sample 1 and 2.
The vertical line represents the distance between these samples.
Similarly, we plot all the steps where we merged the clusters and finally, we get a dendrogram like this: We can clearly visualize the steps of hierarchical clustering.
More the distance of the vertical lines in the dendrogram, more the distance between those clusters.
Now, we can set a threshold distance and draw a horizontal line (Generally, we try to set the threshold in such a way that it cuts the tallest vertical line).
Let’s set this threshold as 12 and draw a horizontal line: The number of clusters will be the number of vertical lines which are being intersected by the line drawn using the threshold.
In the above example, since the red line intersects 2 vertical lines, we will have 2 clusters.
One cluster will have a sample (1,2,4) and the other will have a sample (3,5).
Pretty straightforward, right?.This is how we can decide the number of clusters using a dendrogram in Hierarchical Clustering.
In the next section, we will implement hierarchical clustering which will help you to understand all the concepts that we have learned in this article.
Solving the Wholesale Customer Segmentation problem using Hierarchical Clustering Time to get our hands dirty in Python!.We will be working on a wholesale customer segmentation problem.
You can download the dataset using this link.
The data is hosted on the UCI Machine Learning repository.
The aim of this problem is to segment the clients of a wholesale distributor based on their annual spending on diverse product categories, like milk, grocery, region, etc.
Let’s explore the data first and then apply Hierarchical Clustering to segment the clients.
We will first import the required libraries: View the code on Gist.
Load the data and look at the first few rows: View the code on Gist.
There are multiple product categories – Fresh, Milk, Grocery, etc.
The values represent the number of units purchased by each client for each product.
Our aim is to make clusters from this data that can segment similar clients together.
We will, of course, use Hierarchical Clustering for this problem.
But before applying Hierarchical Clustering, we have to normalize the data so that the scale of each variable is the same.
Why is this important?.Well, if the scale of the variables is not the same, the model might become biased towards the variables with a higher magnitude like Fresh or Milk (refer to the above table).
So, let’s first normalize the data and bring all the variables to the same scale: View the code on Gist.
Here, we can see that the scale of all the variables is almost similar.
Now, we are good to go.
Let’s first draw the dendrogram to help us decide the number of clusters for this particular problem: View the code on Gist.
The x-axis contains the samples and y-axis represents the distance between these samples.
The vertical line with maximum distance is the blue line and hence we can decide a threshold of 6 and cut the dendrogram: View the code on Gist.
We have two clusters as this line cuts the dendrogram at two points.
Let’s now apply hierarchical clustering for 2 clusters: View the code on Gist.
We can see the values of 0s and 1s in the output since we defined 2 clusters.
0 represents the points that belong to the first cluster and 1 represents points in the second cluster.
Let’s now visualize the two clusters: View the code on Gist.
Awesome!.We can clearly visualize the two clusters here.
This is how we can implement hierarchical clustering in Python.
End Notes Hierarchical clustering is a super useful way of segmenting observations.
The advantage of not having to pre-define the number of clusters gives it quite an edge over k-Means.
If you are still relatively new to data science, I highly recommend taking the Applied Machine Learning course.
It is one of the most comprehensive end-to-end machine learning courses you will find anywhere.
Hierarchical clustering is just one of a diverse range of topics we cover in the course.
What are your thoughts on hierarchical clustering?.Do you feel there’s a better way to create clusters using less computational resources?. More details