Editing Cross-validation (statistics) (section)

====Leave-p-out cross-validation====
Leave-''p''-out cross-validation ('''LpO CV''') involves using ''p'' observations as the validation set and the remaining observations as the training set. This is repeated on all ways to cut the original sample on a validation set of ''p'' observations and a training set.<ref>{{cite journal |last1=Celisse |first1=Alain |title=Optimal cross-validation in density estimation with the L<sup>2</sup>-loss |journal=The Annals of Statistics |date=October 2014 |volume=42 |issue=5 |doi=10.1214/14-AOS1240 |arxiv=0811.0802 }}</ref>

LpO cross-validation require training and validating the model <math>C^n_p</math> times, where ''n'' is the number of observations in the original sample, and where <math>C^n_p</math> is the [[binomial coefficient]].  For ''p'' > 1 and for even moderately large ''n'', LpO CV can become computationally infeasible.  For example, with ''n'' = 100 and ''p'' = 30, <math>C^{100}_{30} \approx 3\times 10^{25}.</math>

A variant of LpO cross-validation with p=2 known as leave-pair-out cross-validation has been recommended as a nearly unbiased method for estimating the area under [[ROC curve]] of binary classifiers.<ref>{{cite journal |last1=Airola |first1=Antti |last2=Pahikkala |first2=Tapio |last3=Waegeman |first3=Willem |last4=De Baets |first4=Bernard |last5=Salakoski |first5=Tapio |title=An experimental comparison of cross-validation techniques for estimating the area under the ROC curve |journal=Computational Statistics & Data Analysis |date=April 2011 |volume=55 |issue=4 |pages=1828–1844 |doi=10.1016/j.csda.2010.11.018 }}</ref>