In that case you catch every good investment but incur a significant cost.
Also note that in the example ROC Curve above, Random Forest is able to reduce opportunity cost to nearly zero while Naive Bayes is unable to do the same without incurring maximum cost.
Generally, we will always have to accept at least a few missed opportunities.
Finally, notice that the ROC Curve exhibits the same diminishing returns as the efficient frontier.
This means that each subsequent attempt to reduce missed opportunities will cost you more.
The Buck/CostThere is a tradeoff between reducing missed opportunities (a.
k.
a.
false negatives) and increasing our model’s False Positive Rate (the cost).
Let’s calculate the cost of our previous example first:False Positive Rate = 20 / (40 + 20) = 33%This means that there were 40 + 20 = 60 bad investments and sadly, we got stuck with 20, or 33%, of them.
In our example, these are real negative costs not just missed opportunities.
We invested our money and got hit with losses 20 times, ouch!Let’s dive deeper into the cost vs.
benefit tradeoff represented by the ROC Curve.
Assuming my model has real signal (that it is not just producing random noise), I can use a probability threshold to adjust how often it turns on.
For example, if I set the threshold to 90% (only predict a Good Investment when the model spits out a probability of 90% or more), it would almost never turn on — but when it does, we would expect to make money more often than not.
On the other hand if I set the threshold to 10%, our model would turn on all the time.
And we would invest in plenty of stocks but much more of these (relative to the 90% threshold case) would be money losers.
These money losers are the false positives, the costs we incur in our quest to capture as many good investments (true positives) as we can.
Tying it All TogetherHow to properly tune the threshold of your model is a topic for another day.
It depends heavily on the individual characteristics of your model, the average cost of a False Positive vs.
that of a False Negative, etc.
But the one key I want you to takeaway from all this is that every classification model can be framed as a tradeoff between:Opportunity Cost: The higher the benefit/True Positive Rate of your model’s predictions, the fewer opportunities you miss out on and the lower your opportunity cost.
But when you lower opportunity cost you end up casting a wider net and along with more true positives you also catch more…False Positives: A higher false positive rate means your model takes more actions but is also more frequently wrong when it does turn on.
These false positives are your cost incurred.
And the ROC Curve is a visualization of how effectively a particular model trades off between cost and benefit.
That is why the model with the highest area under the curve, like the efficient frontier, is the model that dominates all others in terms of delivering the most benefit for the amount of cost incurred.
.