Ha!Cross-validationValidation set approachThis is the most basic approach.
It simply involves randomly dividing the dataset into two parts: a training set and a validation set or hold-out set.
The model is fit on the training set and the fitted model is used to make predictions on the validation set.
Validation set schematicAbove is a schematic of the validation set approach.
You have n observations in a dataset, it was randomly split into two parts.
The blue side represents the training set, and the orange side is the validation set.
The numbers simply represent the rows.
Of course, with such a simple approach, there are some drawbacks.
First, the validation test error rate is highly variable depending on which observations are in the training and validation set.
Second, only a small subset of the observations are used to fit the model.
However, we know that statistical methods tend to perform worse when trained on less data.
MSE for the validation set approachAbove, on the left, you see the MSE when the validation set approach was applied only once.
On the right, the process was repeated 10 times.
As you can see, the MSE greatly varies.
This shows the significant variability of the MSE when the validation set approach is used.
Of course, there are methods that address these drawbacks.
Leave-one-out cross-validationThe leave-one-out cross-validation (LOOCV) is a better option than the validation set approach.
Instead of splitting the dataset into two subsets, only one observation is used for validation and the rest is used to fit the model.
LOOCV schematicAbove is a schematic of LOOCV.
As you can see, only one observation is used for validation and the rest is used for training.
The process is then repeated multiple times.
After multiple runs, the error is estimated as:LOOCV estimated errorWhich is simply the mean of the errors of each run.
This method is much better, because it has far less bias, since more observations are used to fit the model.
There is no randomness in the training/validation set splits.
Therefore, we reduce the variability of the MSE, as shown below.
MSE of LOOCVk-fold cross-validationThis approach involves randomly dividing the set of observations into k groups or folds of approximately equal size.
The first fold is treated as a validation set and the model is fit on the remaining folds.
The procedure is then repeated k times, where a different group is treated as the validation set.
k-fold cross-validation schematicHence, you realize that LOOCV is a special case of k-fold cross validation where k is equal to total number of observations n.
However, it is common to set k equal to 5 or 10.
Whereas LOOCV is computationally intensive for large datasets, k-fold is more general and it can be used with any model.
In addition, it often gives more accurate estimates of test error than does LOOCV.
BootstrapBootstrap is an extremely powerful statistical tool used to quantify the uncertainty associated with a given estimator or model.
A statistical software can often output the standard error automatically for some models, such as linear regression.
However, the real power of bootstrap lies in the fact that it can be applied to a wide range of models where the variability is hard to obtain or not output automatically.
To assess and validate your model, the k-fold cross-validation approach is the best option.
It is more computationally efficient and more accurate than LOOCV or the validation set approach.
I personally use that technique often.
As for bootstrap, I haven’t had the chance to apply it yet, since cross-validation is usually enough to assess a model’s performance.
For any questions or suggestions, please write a comment!.Of course, leaving a few claps will greatly motivate me, it’s always appreciated!Stay tuned for more articles and projects in data science.
Cheers!.