Machine Learning Boosting Via Adaptive BoostingUnderstand the most widely used ensemble method that learns from its mistakesFarhad MalikBlockedUnblockFollowFollowingJun 10This article will explain how adaptive boosting works.
Boosting is an ensemble technique.
Boosting enables us to implement a strong model by combining a number of weak models together.
We will concentrate on the Adaptive Boosting technique in this article.
It’s A Not To Miss Ensemble Machine Learning ConceptPhoto by Alvaro Reyes on UnsplashAdditionally, I will demonstrate how we can utilise Adaptive Boosting in Python.
Adaptive Boosting Learns From The Previous MistakesThe concept of Adaptive Boost revolves around correcting previous classifier mistakes.
Each classifier gets trained on the sample set and learns to predict.
The mis-classification errors are then fed into the next classifier in the chain and are used to correct the mistakes until the final model predicts accurate results.
Photo by Josh Riemer on UnsplashAdaptive boosting is also known as AdaBoostIn my article below, I explained how meta classifier ensemble works whereby it is a solution that combines a large number of classifiers to produce more accurate and robust predictions than the predictions by each individual classifier:Let’s Talk About Machine Learning Ensemble Learning In PythonBuild Better Predictive Models By Efficiently Combining Classifiers Into A Meta-Classifiertowardsdatascience.
comAdditionally, the article below provided an overview of one of most popular ensemble techniques known as Bagging Or Bootstrap Aggregating.
Bagging technique can be an effective approach to reduce the variance of a model, to prevent over-fitting and to increase the accuracy of unstable models.
Machine Learning BaggingExplaining How Accuracy Of Unstable Machine Learning Models Can Be Improved Via Bagging Techniquemedium.
comRandom Samples Are Replaced In BaggingThe samples within bagging technique are bootstrapped each time when the model is trained.
When the samples are chosen, they are used to train and validate the predictions.
The samples are then replaced back into the training set.
The samples are selected at random.
This technique is known as bagging.
Photo by Jeremy Bishop on UnsplashThis is slightly different than AdaBoost whereby the samples are trained by a classifier and then the errors are used to correct the mistakes.
How Does Adaptive Boosting WorkThe core concept of boosting focuses on those specific training samples that are hard to classify.
These training samples are known to be misclassified by the classifiers.
These models (classifers) are known as the weak classifiers.
When a weak-classifier misclassifies a training sample, the algorithm then uses these very samples to improve the performance of the ensemble.
Photo by JESHOOTS.
COM on UnsplashIn contrast to bagging, samples drawn from the training dataset are not replaced back into the training set.
AdaBoost Boosts ClassifiersThe entire training set is fed in the AdaBoost.
Eventually a strong classifier is built that learns from the mistakes of the previous weak learners in the ensemble.
There could be a number of iterations (weak classifiers) combined to create a strong classifier.
The image below illustrates how AdaBoost works for 3 iterations.
Let’s consider that there are three weak classifiers named WC1, WC2 and WC3.
These weak classifiers can be based on the Decision Trees algorithm.
Each iteration is used to update the weights by comparing the predicted and actual training data.
The error rate is known as the misclassification rate.
AdaBoost can over-fit on the training set.
Having said that, boosting algorithms are known to decrease bias and variance when they are compared to the bagging models.
Implement AdaBoost In PythonSci-kit learn offers a module that we can use to gain benefits of the Adaptive Boosting algorithm:from sklearn.
ensemble import AdaBoostClassifiertree = DecisionTreeClassifier(criterion='entropy')booster = AdaBoostClassifier(base_estimator=tree)booster = booster.
fit(x, y)The accuracy of the algorithm can be far superior than the accuracy attained by the decision tree alone.
Complexity & Computationally IntensiveIf you analyse the decision boundaries, known as stumps, that are computed by the Adaptive Boosting algorithm when compared with the Decision trees, you will note that the decision boundaries computed by AdaBoost can be very sophisticated.
Although this can help us implement a strong predictive model but the ensemble learning increases the computational complexity compared to individual classifiers.
Photo by Maarten van den Heuvel on UnsplashSummaryThis article highlighted one of the most widely used ensemble technique known as Adaptive Boosting (AdaBoost).
Boosting enables us to implement a strong model by combining a number of weak models together.
Although AdaBoost can produce sophisticated and complex decision boundaries but the algorithm is computationally intensive and is slower than using the individual classifier.
Hope it helps.
.