Imbalanced classification are those prediction tasks where the distribution of examples across class labels is not equal.
Most imbalanced classification examples focus on binary classification tasks, yet many of the tools and techniques for imbalanced classification also directly support multi-class classification problems.
In this tutorial, you will discover how to use the tools of imbalanced classification with a multi-class dataset.
After completing this tutorial, you will know:Discover SMOTE, one-class classification, cost-sensitive learning, threshold moving, and much more in my new book, with 30 step-by-step tutorials and full Python source code.
Let’s get started.
Multi-Class Imbalanced ClassificationPhoto by istolethetv, some rights reserved.
This tutorial is divided into three parts; they are:In this tutorial, we will focus on the standard imbalanced multi-class classification problem referred to as “Glass Identification” or simply “glass.
”The dataset describes the chemical properties of glass and involves classifying samples of glass using their chemical properties as one of six classes.
The dataset was credited to Vina Spiehler in 1987.
Ignoring the sample identification number, there are nine input variables that summarize the properties of the glass dataset; they are:The chemical compositions are measured as the weight percent in corresponding oxide.
There are seven types of glass listed; they are:Float glass refers to the process used to make the glass.
There are 214 observations in the dataset and the number of observations in each class is imbalanced.
Note that there are no examples for class 4 (non-float processed vehicle windows) in the dataset.
Although there are minority classes, all classes are equally important in this prediction problem.
The dataset can be divided into window glass (classes 1-4) and non-window glass (classes 5-7).
There are 163 examples of window glass and 51 examples of non-window glass.
Another division of the observations would be between float processed glass and non-float processed glass, in the case of window glass only.
This division is more balanced.
You can learn more about the dataset here:No need to download the dataset; we will download it automatically as part of the worked examples.
Below is a sample of the first few rows of the data.
We can see that all inputs are numeric and the target variable in the final column is the integer encoded class label.
You can learn more about how to work through this dataset as part of a project in the tutorial:Now that we are familiar with the glass multi-class classification dataset, let’s explore how we can use standard imbalanced classification tools with it.
Take my free 7-day email crash course now (with sample code).
Click to sign-up and also get a free PDF Ebook version of the course.
Download Your FREE Mini-CourseOversampling refers to copying or synthesizing new examples of the minority classes so that the number of examples in the minority class better resembles or matches the number of examples in the majority classes.
Perhaps the most widely used approach to synthesizing new examples is called the Synthetic Minority Oversampling TEchnique, or SMOTE for short.
This technique was described by Nitesh Chawla, et al.
in their 2002 paper named for the technique titled “SMOTE: Synthetic Minority Over-sampling Technique.
”You can learn more about SMOTE in the tutorial:The imbalanced-learn library provides an implementation of SMOTE that we can use that is compatible with the popular scikit-learn library.
First, the library must be installed.
We can install it using pip as follows:sudo pip install imbalanced-learnWe can confirm that the installation was successful by printing the version of the installed library:Running the example will print the version number of the installed library; for example:Before we apply SMOTE, let’s first load the dataset and confirm the number of examples in each class.
Running the example first downloads the dataset and splits it into train and test sets.
The number of rows in each class is then reported, confirming that some classes, such as 0 and 1, have many more examples (more than 70) than other classes, such as 3 and 4 (less than 15).
A bar chart is created providing a visualization of the class breakdown of the dataset.
This gives a clearer idea that classes 0 and 1 have many more examples than classes 2, 3, 4 and 5.
Histogram of Examples in Each Class in the Glass Multi-Class Classification DatasetNext, we can apply SMOTE to oversample the dataset.
By default, SMOTE will oversample all classes to have the same number of examples as the class with the most examples.
In this case, class 1 has the most examples with 76, therefore, SMOTE will oversample all classes to have 76 examples.
The complete example of oversampling the glass dataset with SMOTE is listed below.
Running the example first loads the dataset and applies SMOTE to it.
The distribution of examples in each class is then reported, confirming that each class now has 76 examples, as we expected.
A bar chart of the class distribution is also created, providing a strong visual indication that all classes now have the same number of examples.
Histogram of Examples in Each Class in the Glass Multi-Class Classification Dataset After Default SMOTE OversamplingInstead of using the default strategy of SMOTE to oversample all classes to the number of examples in the majority class, we could instead specify the number of examples to oversample in each class.
For example, we could oversample to 100 examples in classes 0 and 1 and 200 examples in remaining classes.
This can be achieved by creating a dictionary that maps class labels to the number of desired examples in each class, then specifying this via the “sampling_strategy” argument to the SMOTE class.
Tying this together, the complete example of using a custom oversampling strategy for SMOTE is listed below.
Running the example creates the desired sampling and summarizes the effect on the dataset, confirming the intended result.
Note: you may see warnings that can be safely ignored for the purposes of this example, such as:A bar chart of the class distribution is also created confirming the specified class distribution after data sampling.
Histogram of Examples in Each Class in the Glass Multi-Class Classification Dataset After Custom SMOTE OversamplingNote: when using data sampling like SMOTE, it must only be applied to the training dataset, not the entire dataset.
I recommend using a Pipeline to ensure that the SMOTE method is correctly used when evaluating models and making predictions with models.
You can see an example of the correct usage of SMOTE in a Pipeline in this tutorial:Most machine learning algorithms assume that all classes have an equal number of examples.
This is not the case in multi-class imbalanced classification.
Algorithms can be modified to change the way learning is performed to bias towards those classes that have fewer examples in the training dataset.
This is generally called cost-sensitive learning.
For more on cost-sensitive learning, see the tutorial:The RandomForestClassifier class in scikit-learn supports cost-sensitive learning via the “class_weight” argument.
By default, the random forest class assigns equal weight to each class.
We can evaluate the classification accuracy of the default random forest class weighting on the glass imbalanced multi-class classification dataset.
The complete example is listed below.
Running the example evaluates the default random forest algorithm with 1,000 trees on the glass dataset using repeated stratified k-fold cross-validation.
The mean and standard deviation classification accuracy are reported at the end of the run.
Your specific results may vary given the stochastic nature of the learning algorithm, the evaluation procedure, and differences in precision across machines.
Try running the example a few times.
In this case, we can see that the default model achieved a classification accuracy of about 79.
6 percent.
We can specify the “class_weight” argument to the value “balanced” that will automatically calculates a class weighting that will ensure each class gets an equal weighting during the training of the model.
Tying this together, the complete example is listed below.
Running the example reports the mean and standard deviation classification accuracy of the cost-sensitive version of random forest on the glass dataset.
Your specific results may vary given the stochastic nature of the learning algorithm, the evaluation procedure, and differences in precision across machines.
Try running the example a few times.
In this case, we can see that the default model achieved a lift in classification accuracy over the cost-insensitive version of the algorithm, with 80.
2 percent classification accuracy vs.
79.
6 percent.
The “class_weight” argument takes a dictionary of class labels mapped to a class weighting value.
We can use this to specify a custom weighting, such as a default weighting for classes 0 and 1.
0 that have many examples and a double class weighting of 2.
0 for the other classes.
Tying this together, the complete example of using a custom class weighting for cost-sensitive learning on the glass multi-class imbalanced classification problem is listed below.
Running the example reports the mean and standard deviation classification accuracy of the cost-sensitive version of random forest on the glass dataset with custom weights.
Your specific results may vary given the stochastic nature of the learning algorithm, the evaluation procedure, and differences in precision across machines.
Try running the example a few times.
In this case, we can see that we achieved a further lift in accuracy from about 80.
2 percent with balanced class weighting to 80.
8 percent with a more biased class weighting.
This section provides more resources on the topic if you are looking to go deeper.
In this tutorial, you discovered how to use the tools of imbalanced classification with a multi-class dataset.
Specifically, you learned:Do you have any questions? Ask your questions in the comments below and I will do my best to answer.
with just a few lines of python codeDiscover how in my new Ebook: Imbalanced Classification with PythonIt provides self-study tutorials and end-to-end projects on: Performance Metrics, Undersampling Methods, SMOTE, Threshold Moving, Probability Calibration, Cost-Sensitive Algorithms and much more.
.