How do we know our model is better than any other model?This blog post will explain the metrics used to test the robustness of the model but before we discuss them, we will dive into the most basic components of a confusion matrix.
In the first iteration, we perhaps want to build a base model, a logistic regression model is a good choice.
The model classifies each day into 1 or 0 i.
e.
(rain or no rain), but remember our logistic regression model will give us probabilities of 1 and 0, e.
g.
for a particular day, it might predict rain with 90% certainty and no rain with 10%.
It’s upon us to decide which percentage threshold we will consider as 1.
We may classify a day as a rainy day whenever our model is at least 50% confident of predicting rain.
In this example 50% threshold is arbitrary, it depends on the problemWe now have a foundation to discuss the confusion matrix, but before we do, please find its definition from Wikipedia which I have further simplified.
In classification problems, a confusion matrix, also known as an error matrix, is a specific table layout that allows visualization of the performance of an algorithm.
Most commonly, the ROC curve is built as a visualization to understand the performance of the model, but in this post, we will not discuss the ROC, but the metrics derived from the confusion matrix, which allows us to get those visualizations.
“Post on ROC curve is reserved for another day.
”The terminology you will find most often used for confusion matrix has 2 components.
i.
e.
positive class and negative class, because we are interested in predicting rain thus, rain = positive class and no rain= negative class.
From these 2 components, we derive 4 terms.
TP = True Positives, FN = False Negatives, TN = True Negatives, FP = False PositivesTo understand the confusion matrix, It’s extremely important to understand the 4 components.
Let’s see how these 4 components will be evaluated for rain prediction.
note: please read, ^ = prediction, i.
e.
P(predicted value|actual value)TP = True PositiveWhen our model is at least 50% confident of rainfall, we will predict 1 (rain) as the outcome and in fact, if it turns its true, then it's a true-positive prediction.
When we predict the positive class correctly, it’s called True Positive.
TN = True NegativeImagine the negative case, when our model is at least 50% confident of no rain, we predict 0(no rain) as the outcome and in fact, if it is true, then it’s a true negative prediction.
When we predict the negative class correctly, it’s called True Negative.
Things get interesting for the next 2 types of predictions.
What would happen if we are wrong?FP = False PositiveImagine, our model predicts it will rain, but to our surprise, no sign of rain.
i.
e.
our model got it wrong.
It predicted the positive class (rain) however, the correct value was the negative class, i.
e.
it was a day with no rain.
When we predict the positive class incorrectly, it’s called False Positive.
Next is the 4th component of the confusion matrix.
FN = False NegativeCompare the image for false positive and false negative, you will find events are rearranged, In this scenario, our model predicted the negative class (no rain) but the correct value was a positive class.
When we predict the negative class incorrectly, It’s called False Negative.
Below picture is the best example of a False outcome (wrong prediction).
(Please ignore if the patient in the below picture looks like Neil deGrasse Tyson)Illustration of False Positive and False NegativeDeriving metrics from Confusion MatrixRemember this: Positive Class(P) = TP + FN and Negative Class(N) = TN + FPLet’s predict rain for 50 days and then arrange our predictions against their correct values in a square matrix.
Confusion Matrix1|TPR (True Positive Rate)TPR = True Positive RateThe True Positive rate is given by TPR = TP/P, i.
e.
True positive values over all the positive values.
Remember, P= TP+FN.
Thus, TPR = TP/(TP+FN).
This metric is also known as Sensitivity in statistics or Recall in machine learning.
This metric helps us understand how well we are predicting our positive case.
In our example TPR = 20/(20+5) = 0.
8, i.
e.
our model correctly predicts rain 80% of the time, when it rained.
2|TNR (True Negative Rate)TNR = True Negative RateThe True Negative rate is given by TNR = TN/N.
i.
e.
True negative values over all the negative values.
Remember, N= TN+FP.
Thus, TNR = TN/(TN+FP).
This metric is also known as Specificity in statistics.
This metric helps us understand how well we are predicting our negative case.
In our example TNR = 15/(15+10) = 0.
6, i.
e.
our model correctly predicts no rain 60% of the time, when there was no rain.
quick revision: TPR = Sensitivity/Recall, TNR = Specificity3 |False Positive RateFPR = False Positive RateWe might be interested to know, how many times the model predicted the positive class or rain but there was no rain for that day.
It’s defined as FP/N, i.
e.
False positive values over all the negative values.
If we use the fact that N = TN + FP then, FPR = FP/(TN+FP), In our example, FPR = 10/(15+10) = 0.
4.
i.
e our model incorrectly predicted rain 40% of the time, when there was no rain.
4 |Accuracyaccuracy = TPR + FNRWhen we want to know how well our model predicts the positive class & the negative class, we turn to accuracy.
It’s defined as (TP + TN)/(P + N).
In the above example, our model’s accuracy is (20 + 15)/(20+5+15+10) = 0.
7, i.
e.
our model correctly predicts the rain and no rain 70% of the time.
5 |Error Rateerror rate or misclassification rate = FPR + FNRThe Error rate is also called the misclassification rate.
It’s the opposite of accuracy or if represented mathematically the error rate = (1- accuracy) = (FP + FN)/(P+N).
The Error rate for our rain example is (1 – 0.
7) = 0.
3, i.
e.
we 30% percent of the time we didn't get the correct prediction.
I will not define the FNR (False Negative Rate) because I believe if you are new to the confusion matrix you can easily get confused, this post already has 6 metrics.
However, I believe after learning from this post you will figure it out yourself.
6 |PPV (Positive Predicted Value)PPV = Precision, P(actual value|predicted value)When we want to know, how many correct predictions we made for the positive class from all the predictions we made for the positive class, then we know we are looking for precision in our model’s prediction.
Hence, the Positive Prediction Value is also called Precision in machine learning.
It is defined as TP/(TP+FP).
For the example above, our model’s PPV or Precision = 20/(20+10) = 0.
66…, i.
e.
our model predicted rain correctly 66 % of the time whenever it predicted rain.
ConclusionTP, TN, FP, FN are the building blocks of all the classification metrics that are most commonly used.
It’s mostly the False in the positive or True in the Negative etc.
that cause confusion if you read them or the first time, so don't be disheartened if you struggle a bit, I hope this blog post helped you to understand the confusion matrix.
.