Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have found the following definitions, but I don't really see the difference.
cross validation Method for testing classification and prediction models. The data are randomly split into N partitions (typically N=10) and then N times a model is created from N-1 partitions and tested on the "holdout" data.
Leave one out Every data point gets to be in a test set exactly once, and gets to be in a training set k-1 times.
Assuming your dataset includes $k$ samples:
In cross-validation, there are $N$ partitions, and the test split for each partition will have size $\frac{k}{N}$.
Leave-one-out validation is a special type of cross-validation where $N = k$. You can think of this as taking cross-validation to its extreme, where we set the number of partitions to its maximum possible value. In leave-one-out validation, the test split will have size $\frac{k}{k} = 1$
It's easy to visualize the difference. Here's two figures which contrast cross-validation and leave-one-out. In these figures, each sample in the dataset is represented by a colored circle. The training set is represented by the green circles, and the testing set is represented by the yellow circles.
5-fold cross validation:
Leave-one-out cross validation:
Consider a data set in which you have users and want to predict something about them based on some values. Each user might have many entries in the DS.
In cross validation, data is partitioned randomly so different users entries can fall into the same partition or some partitions might have no information about some user user, like below:
Dataset -> Partition
user1; somevalues -> partition 1
user1; someOthervalues -> partition 2
user2; somevalues -> partition 2
user1; more_on_user1 -> partition 1
user2; someMoreValues -> partition 2
user3; somevalues -> partition 1
etc...
partition 1 knows nothing about user2. Which might not be importan if you're assuming that behaviour is global an not particular to the user
Whereas leave-one-out will ensure that the test set has one example of each user. This is very common in recommendation systems in which for each user you leave one rating out, because a metric tested in all users is more reliable (in the context of rec.sys) than a metric that might skip many of them.
