0
$\begingroup$

I do not get why in For cross validation should I use training set, or whole dataset? the responses say that cross validation must be done exclusively on training set. Doesn't the methods (for example in scikit) make for each split an independent train-set procedure? Isn't that the whole point of CV?

$\endgroup$

1 Answer 1

1
$\begingroup$

Cross-validation can be used on two levels.

  1. When you want to evaluate an estimator $h$ by estimating its risk $\mathbb{E}[L(h(x),y)]$ based on some loss function $L$, you need independent samples to compute the estimator $h$ and to compute the expectation. Then you can use cross-validation on all the data to get a better estimate of the risk than just using a single data split. Typically, this is done for a final analysis of a learning algorithm to publish its performance in a paper or a report.

  2. In many cases learning algorithms also need to estimate the risk during their training process for hyper-parameter tuning (for example selecting the regularization parameter for a linear regression model), for which they can also use cross-validation. Then cross-validation is only applied to the training data as it is part of the training process.

The other issue raised in the linked post do not seem to me specific to cross-validation only. If you conduct your research on the same data for years designing and tuning many algorithms on it, then you will likely overfit that data irrespective whether you use cross-validation or any other data splitting method.

Data leakage typically becomes an issue when the samples are not independent, which often happens for time series data. Then you might have to discard samples to avoid leakage. But this issue again applies to all data splitting methods, not just cross-validation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.