Machine learning predictive modeling performance is only as good as your data, and your data is only as good as the way you prepare it for modeling.
The most common approach to data preparation is to study a dataset and review the expectations of a machine learning algorithms, then carefully choose the most appropriate data preparation techniques to transform the raw data to best meet the expectations of the algorithm.
This is slow, expensive, and requires a vast amount of expertise.
An alternative approach to data preparation is to grid search a suite of common and commonly useful data preparation techniques to the raw data.
This is an alternative philosophy for data preparation that treats data transforms as another hyperparameter of the modeling pipeline to be searched and tuned.
This approach requires less expertise than the traditional manual approach to data preparation, although it is computationally costly.
The benefit is that it can aid in the discovery of non-intuitive data preparation solutions that achieve good or best performance for a given predictive modeling problem.
In this tutorial, you will discover how to use the grid search approach for data preparation with tabular data.
After completing this tutorial, you will know:Discover data cleaning, feature selection, data transforms, dimensionality reduction and much more in my new book, with 30 step-by-step tutorials and full Python source code.
Let’s get started.
How to Grid Search Data Preparation TechniquesPhoto by Wall Boat, some rights reserved.
This tutorial is divided into three parts; they are:Data preparation can be challenging.
The approach that is most often prescribed and followed is to analyze the dataset, review the requirements of the algorithms, and transform the raw data to best meet the expectations of the algorithms.
This can be effective but is also slow and can require deep expertise with data analysis and machine learning algorithms.
An alternative approach is to treat the preparation of input variables as a hyperparameter of the modeling pipeline and to tune it along with the choice of algorithm and algorithm configurations.
This might be a data transform that “should not work” or “should not be appropriate for the algorithm” yet results in good or great performance.
Alternatively, it may be the absence of a data transform for an input variable that is deemed “absolutely required” yet results in good or great performance.
This can be achieved by designing a grid search of data preparation techniques and/or sequences of data preparation techniques in pipelines.
This may involve evaluating each on a single chosen machine learning algorithm, or on a suite of machine learning algorithms.
The benefit of this approach is that it always results in suggestions of modeling pipelines that give good relative results.
Most importantly, it can unearth the non-obvious and unintuitive solutions to practitioners without the need for deep expertise.
We can explore this approach to data preparation with a worked example.
Before we dive into a worked example, let’s first select a standard dataset and develop a baseline in performance.
Take my free 7-day email crash course now (with sample code).
Click to sign-up and also get a free PDF Ebook version of the course.
Download Your FREE Mini-CourseIn this section, we will first select a standard machine learning dataset and establish a baseline in performance on this dataset.
This will provide the context for exploring the grid search method of data preparation in the next section.
We will use the wine classification dataset.
This dataset has 13 input variables that describe the chemical composition of samples of wine and requires that the wine be classified as one of three types.
You can learn more about the dataset here:No need to download the dataset as we will download it automatically as part of our worked examples.
Open the dataset and review the raw data.
The first few rows of data are listed below.
We can see that it is a multi-class classification predictive modeling problem with numerical input variables, each of which has different scales.
The example below loads the dataset and splits it into the input and output columns, then summarizes the data arrays.
Running the example, we can see that the dataset was loaded correctly and that there are 179 rows of data with 13 input variables and a single target variable.
Next, let’s evaluate a model on this dataset and establish a baseline in performance.
We can establish a baseline in performance on the wine classification task by evaluating a model on the raw input data.
In this case, we will evaluate a logistic regression model.
First, we can define a function to load the dataset and perform some minimal data preparation to ensure the inputs are numeric and the target is label encoded.
We will evaluate the model using the gold standard of repeated stratified k-fold cross-validation with 10 folds and three repeats.
We can then call the function to load the dataset, define our model, then evaluate it, reporting the mean and standard deviation accuracy.
Tying this together, the complete example of evaluating a logistic regression model on the raw wine classification dataset is listed below.
Running the example evaluates the model performance and reports the mean and standard deviation classification accuracy.
Your results may vary given the stochastic nature of the learning algorithm, the evaluation procedure, and differences in precision across machines.
Try running the example a few times.
In this case, we can see that the logistic regression model fit on the raw input data achieved the average classification accuracy of about 95.
3 percent, providing a baseline in performance.
Next, let’s explore whether we can improve the performance using the grid-search-based approach to data preparation.
In this section, we can explore whether we can improve performance using the grid search approach to data preparation.
The first step is to define a series of modeling pipelines to evaluate, where each pipeline defines one (or more) data preparation techniques and ends with a model that takes the transformed data as input.
We will define a function to create these pipelines as a list of tuples, where each tuple defines the short name for the pipeline and the pipeline itself.
We will evaluate a range of different data scaling methods (e.
g.
MinMaxScaler and StandardScaler), distribution transforms (QuantileTransformer and KBinsDiscretizer), as well as dimensionality reduction transforms (PCA and SVD).
We can then call this function to get the list of transforms, then enumerate each, evaluating it and reporting the performance along the way.
At the end of the run, we can create a box and whisker plot for each set of scores and compare the distributions of results visually.
Tying this together, the complete example of grid searching data preparation techniques on the wine classification dataset is listed below.
Running the example evaluates the performance of each pipeline and reports the mean and standard deviation classification accuracy.
Your results may vary given the stochastic nature of the learning algorithm, the evaluation procedure, and differences in precision across machines.
Try running the example a few times.
In this case, we can see that standardizing the input variables and using a quantile transform both achieves the best result with a classification accuracy of about 98.
7 percent, an improvement over the baseline with no data preparation that achieved a classification accuracy of 95.
3 percent.
You can add your own modeling pipelines to the get_pipelines() function and compare their result.
Can you get better results? Let me know in the comments below.
A figure is created showing box and whisker plots that summarize the distribution of classification accuracy scores for each data preparation technique.
We can see that the distribution of scores for the standardization and quantile transforms are compact and very similar and have an outlier.
We can see that the spread of scores for the other transforms is larger and skewing down.
The results may suggest that standardizing the dataset is probably an important step in the data preparation and related transforms, such as the quantile transform, and perhaps even the power transform may offer benefits if combined with standardization by making one or more input variables more Gaussian.
Box and Whisker Plot of Classification Accuracy for Different Data Transforms on the Wine Classification DatasetWe can also explore sequences of transforms to see if they can offer a lift in performance.
For example, we might want to apply RFE feature selection after the standardization transform to see if the same or better results can be used with fewer input variables (e.
g.
less complexity).
We might also want to see if a power transform preceded with a data scaling transform can achieve good performance on the dataset as we believe it could given the success of the quantile transform.
The updated get_pipelines() function with sequences of transforms is provided below.
Tying this together, the complete example is listed below.
Running the example evaluates the performance of each pipeline and reports the mean and standard deviation classification accuracy.
Your results may vary given the stochastic nature of the learning algorithm, the evaluation procedure, and differences in precision across machines.
Try running the example a few times.
In this case, we can see that the standardization with feature selection offers an additional lift in accuracy from 98.
7 percent to 98.
9 percent, although the data scaling and power transform do not offer any additional benefit over the quantile transform.
A figure is created showing box and whisker plots that summarize the distribution of classification accuracy scores for each data preparation technique.
We can see that the distribution of results for both pipelines of transforms is compact with very little spread other than outlier.
Box and Whisker Plot of Classification Accuracy for Different Sequences of Data Transforms on the Wine Classification DatasetThis section provides more resources on the topic if you are looking to go deeper.
In this tutorial, you discovered how to use a grid search approach for data preparation with tabular data.
Specifically, you learned:Do you have any questions? Ask your questions in the comments below and I will do my best to answer.
with just a few lines of python codeDiscover how in my new Ebook: Data Preparation for Machine LearningIt provides self-study tutorials with full working code on: Feature Selection, RFE, Data Cleaning, Data Transforms, Scaling, Dimensionality Reduction, and much more.
.