Evaluating the Business Value of Predictive Models in Python and R

By Jurriaan Nagelkerke, Data Science Consultant, and Pieter Marcus, Data Scientist    In this blog we explain four most valuable evaluation plots to assess the business value of a predictive model..This will help you to explain your models business value in laymans terms to non-techies.   ‘…And as we can see clearly on this ROC plot, the sensitivity of the model at the value of 0.2 on one minus the specificity is quite high!.Right?…’.If your fellow business colleagues didnt already wander away during your presentation about your fantastic predictive model, it will definitely push them over the edge when you start talking like this..Such as predictive models to select the best records in a dataset, which can be customers, leads, items, events….Another important reason that these statistics and plots are useless in your business meetings is that they dont help in determining how to apply your predictive model: What percentage of records should we select based on the model?.For now, we focus on explaining to our marketing colleagues how good our predictive model can help them select customers for their term deposit campaign.What just happened?.As the output notes, you can use modelplotpy to evaluate your model(s) from several perspectives:Here, we will keep it simple and evaluate – from a business perspective – how well a selected model will perform in a selected dataset for one target class..The default value for the target class is term deposit ; since we want to focus on customers that do take term deposits, this default is perfect.   Before we throw more code and output at you, lets get you familiar with the plots we so strongly advocate to use to assess a predictive models business value..Observations that belong to the top-10% with highest model probability in a set, are in decile 1 of that set; the next group of 10% with high model probability are decile 2 and finally the 10% observations with the lowest model probability on the target class belong to decile 10.Each of our four plots places the deciles on the x axis and another measure on the y axis..Cumulative gains plotThe cumulative gains plot – often named gains plot – helps you answer the question:When we apply the model and select the best X deciles, what % of the actual target class observations can we expect to target?Hence, the cumulative gains plot visualises the percentage of the target class members you have selected if you would decide to select up until decile X..This is a very important business question, because in most cases, you want to use a predictive model to target a subset of observations – customers, prospects, cases,….Or build a perfect model, that scores all actual target class members with a 100% probability and all the cases that do not belong to the target class with a 0% probability..However, if youre such a wizard, you dont need these plots any way or you should have a careful look at your model – maybe youre cheating?….So, well have to accept we will lose some. What percentage of the actual target class members you do select with your model at a given decile, thats what the cumulative gains plot tells you.. More details

Leave a Reply