Editing Cross-validation (statistics) (section)

==Nested cross-validation==

When cross-validation is used simultaneously for selection of the best set of [[Hyperparameter (machine learning)|hyperparameters]] and for error estimation (and assessment of generalization capacity), a nested cross-validation is required. Many variants exist. At least two variants can be distinguished:

===k*l-fold cross-validation===

This is a truly nested variant which contains an outer loop of ''k'' sets and an inner loop of ''l'' sets. The total data set is split into ''k'' sets. One by one, a set is selected as the (outer) test set and the ''k''&nbsp;-&nbsp;1 other sets are combined into the corresponding outer training set. This is repeated for each of the ''k'' sets. Each outer training set is further sub-divided into ''l'' sets.  One by one, a set is selected as inner test (validation) set and the ''l''&nbsp;-&nbsp;1 other sets are combined into the corresponding inner training set. This is repeated for each of the ''l'' sets. The inner training sets are used to fit model parameters, while the outer test set is used as a validation set to provide an unbiased evaluation of the model fit. Typically, this is repeated for many different hyperparameters (or even different model types) and the validation set is used to determine the best hyperparameter set (and model type) for this inner training set. After this, a new model is fit on the entire outer training set, using the best set of hyperparameters from the inner cross-validation. The performance of this model is then evaluated using the outer test set.

===k-fold cross-validation with validation and test set===

This is a type of k*l-fold cross-validation when ''l''&nbsp;=&nbsp;''k''&nbsp;-&nbsp;1. A single k-fold cross-validation is used with both a [[Training, validation, and test sets|validation and test set]]. The total data set is split into ''k'' sets. One by one, a set is selected as test set. Then, one by one, one of the remaining sets is used as a validation set and the other ''k''&nbsp;-&nbsp;2 sets are used as training sets until all possible combinations have been evaluated. Similar to the k*l-fold cross validation, the training set is used for model fitting and the validation set is used for model evaluation for each of the hyperparameter sets. Finally, for the selected parameter set, the test set is used to evaluate the model with the best parameter set. Here, two variants are possible: either evaluating the model that was trained on the training set or evaluating a new model that was fit on the combination of the training and the validation set.