Hyperparameter optimization refers to performing a search in order to discover the set of specific model configuration arguments that result in the best performance of the model on a specific dataset.
There are many ways to perform hyperparameter optimization, although modern methods, such as Bayesian Optimization, are fast and effective.
The Scikit-Optimize library is an open-source Python library that provides an implementation of Bayesian Optimization that can be used to tune the hyperparameters of machine learning models from the scikit-Learn Python library.
You can easily use the Scikit-Optimize library to tune the models on your next machine learning project.
In this tutorial, you will discover how to use the Scikit-Optimize library to use Bayesian Optimization for hyperparameter tuning.
After completing this tutorial, you will know:Let’s get started.
Scikit-Optimize for Hyperparameter Tuning in Machine LearningPhoto by Dan Nevill, some rights reserved.
This tutorial is divided into four parts; they are:Scikit-Optimize, or skopt for short, is an open-source Python library for performing optimization tasks.
It offers efficient optimization algorithms, such as Bayesian Optimization, and can be used to find the minimum or maximum of arbitrary cost functions.
Bayesian Optimization provides a principled technique based on Bayes Theorem to direct a search of a global optimization problem that is efficient and effective.
It works by building a probabilistic model of the objective function, called the surrogate function, that is then searched efficiently with an acquisition function before candidate samples are chosen for evaluation on the real objective function.
For more on the topic of Bayesian Optimization, see the tutorial:Importantly, the library provides support for tuning the hyperparameters of machine learning algorithms offered by the scikit-learn library, so-called hyperparameter optimization.
As such, it offers an efficient alternative to less efficient hyperparameter optimization procedures such as grid search and random search.
The scikit-optimize library can be installed using pip, as follows:Once installed, we can import the library and print the version number to confirm the library was installed successfully and can be accessed.
The complete example is listed below.
Running the example reports the currently installed version number of scikit-optimize.
Your version number should be the same or higher.
For more installation instructions, see the documentation:Now that we are familiar with what Scikit-Optimize is and how to install it, let’s explore how we can use it to tune the hyperparameters of a machine learning model.
First, let’s select a standard dataset and a model to address it.
We will use the ionosphere machine learning dataset.
This is a standard machine learning dataset comprising 351 rows of data with three numerical input variables and a target variable with two class values, e.
g.
binary classification.
Using a test harness of repeated stratified 10-fold cross-validation with three repeats, a naive model can achieve an accuracy of about 64 percent.
A top performing model can achieve accuracy on this same test harness of about 94 percent.
This provides the bounds of expected performance on this dataset.
The dataset involves predicting whether measurements of the ionosphere indicate a specific structure or not.
You can learn more about the dataset here:No need to download the dataset; we will download it automatically as part of our worked examples.
The example below downloads the dataset and summarizes its shape.
Running the example downloads the dataset and splits it into input and output elements.
As expected, we can see that there are 351 rows of data with 34 input variables.
We can evaluate a support vector machine (SVM) model on this dataset using repeated stratified cross-validation.
We can report the mean model performance on the dataset averaged over all folds and repeats, which will provide a reference for model hyperparameter tuning performed in later sections.
The complete example is listed below.
Running the example first loads and prepares the dataset, then evaluates the SVM model on the dataset.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision.
Consider running the example a few times and compare the average outcome.
In this case, we can see that the SVM with default hyperparameters achieved a mean classification accuracy of about 83.
7 percent, which is skillful and close to the top performance on the problem of 94 percent.
Next, let’s see if we can improve performance by tuning the model hyperparameters using the scikit-optimize library.
The Scikit-Optimize library can be used to tune the hyperparameters of a machine learning model.
We can achieve this manually by using the Bayesian Optimization capabilities of the library.
This requires that we first define a search space.
In this case, this will be the hyperparameters of the model that we wish to tune, and the scope or range of each hyperparameter.
We will tune the following hyperparameters of the SVM model:For the numeric hyperparameters C and gamma, we will define a log scale to search between a small value of 1e-6 and 100.
Degree is an integer and we will search values between 1 and 5.
Finally, the kernel is a categorical variable with specific named values.
We can define the search space for these four hyperparameters, a list of data types from the skopt library, as follows:Note the data type, the range, and the name of the hyperparameter specified for each.
We can then define a function that will be called by the search procedure.
This is a function expected by the optimization procedure later and takes a model and set of specific hyperparameters for the model, evaluates it, and returns a score for the set of hyperparameters.
In our case, we want to evaluate the model using repeated stratified 10-fold cross-validation on our ionosphere dataset.
We want to maximize classification accuracy, e.
g.
find the set of model hyperparameters that give the best accuracy.
By default, the process minimizes the score returned from this function, therefore, we will return one minus the accuracy, e.
g.
perfect skill will be (1 – accuracy) or 0.
0, and the worst skill will be 1.
0.
The evaluate_model() function below implements this and takes a specific set of hyperparameters.
Next, we can execute the search by calling the gp_minimize() function and passing the name of the function to call to evaluate each model and the search space to optimize.
The procedure will run until it converges and returns a result.
The result object contains lots of details, but importantly, we can access the score of the best performing configuration and the hyperparameters used by the best forming model.
Tying this together, the complete example of manually tuning the hyperparameters of an SVM on the ionosphere dataset is listed below.
Running the example may take a few moments, depending on the speed of your machine.
You may see some warning messages that you can safely ignore, such as:At the end of the run, the best-performing configuration is reported.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision.
Consider running the example a few times and compare the average outcome.
In this case, we can see that configuration, reported in order of the search space list, was a modest C value, a RBF kernel, a degree of 2 (ignored by the RBF kernel), and a modest gamma value.
Importantly, we can see that the skill of this model was approximately 94.
7 percent, which is a top-performing modelThis is not the only way to use the Scikit-Optimize library for hyperparameter tuning.
In the next section, we can see a more automated approach.
The Scikit-Learn machine learning library provides tools for tuning model hyperparameters.
Specifically, it provides the GridSearchCV and RandomizedSearchCV classes that take a model, a search space, and a cross-validation configuration.
The benefit of these classes is that the search procedure is performed automatically, requiring minimal configuration.
Similarly, the Scikit-Optimize library provides a similar interface for performing a Bayesian Optimization of model hyperparameters via the BayesSearchCV class.
This class can be used in the same way as the Scikit-Learn equivalents.
First, the search space must be defined as a dictionary with hyperparameter names used as the key and the scope of the variable as the value.
We can then define the BayesSearchCV configuration taking the model we wish to evaluate, the hyperparameter search space, and the cross-validation configuration.
We can then execute the search and report the best result and configuration at the end.
Tying this together, the complete example of automatically tuning SVM hyperparameters using the BayesSearchCV class on the ionosphere dataset is listed below.
Running the example may take a few moments, depending on the speed of your machine.
You may see some warning messages that you can safely ignore, such as:At the end of the run, the best-performing configuration is reported.
Note: Your results may vary given the stochastic nature of the algorithm or evaluation procedure, or differences in numerical precision.
Consider running the example a few times and compare the average outcome.
In this case, we can see that the model performed above top-performing models achieving a mean classification accuracy of about 95.
2 percent.
The search discovered a large C value, an RBF kernel, and a small gamma value.
This provides a template that you can use to tune the hyperparameters on your machine learning project.
This section provides more resources on the topic if you are looking to go deeper.
In this tutorial, you discovered how to use the Scikit-Optimize library to use Bayesian Optimization for hyperparameter tuning.
Specifically, you learned:Do you have any questions? Ask your questions in the comments below and I will do my best to answer.
with just a few lines of scikit-learn codeLearn how in my new Ebook: Machine Learning Mastery With PythonCovers self-study tutorials and end-to-end projects like: Loading data, visualization, modeling, tuning, and much more.
Skip the Academics.
Just Results.
.