leave one pair out cross validation

Question

I am trying to train and validate my datasets which contains 17 datasets. I have divided them as 15 for training and 2 for validation. In the process, I train on 15 datasets and use the generated model to predict the results on the remaining 2 datasets. This process is called leave out validation in my understanding. Irrespective of the classifier I use (SVM, optimizable SVM, knn, optimizable KNN), I always get a very high training accuracy closer to 90-100%. The validation accuracy is comparatively poorer closer to 50-60 %. The datasets in the validation set will be a part of training in some runs. In this case, I can not understand if they are doing so good in the training why the validation results are so bad.?

Answer 1

[edited after clarification by OP]

I can not understand if they are doing so good in the training why the validation results are so bad.?

This is certainly due to major differences between the datasets. Supervised learning relies on the assumption that both the training set and the test set are samples from the same population (i.e. they follow the same distribution). In your case the model assumes that the 15 datasets used as training data are a representative sample of "the" data distribution. If the two datasets used as test set have very different distributions, then what the model learned is simply not applicable to the test set.

I think this is a form of overfitting: the model learns some details of the instances instead of the true patterns which matter for the target variable. These details actually happen in the training set, so the performance is good on it. But they don't happen (at least not in the same way) in the validation set, so the true performance is poor.

Note: your evaluation setting is similar to leave one out (LOO) cross-validation, but this is not LOO because LOO means leaving a single instance as test set.