3
$\begingroup$

Let's say we have model M1 and model M2 that we want to compare. When we do 5-fold (say) cross validation, would the correct method to be to partition the data into F1, F2, F3, F4, and F5 and then run both models through those folds? Then would the way to assess if M2 outperforms M1 be to do a paired t-test?

I'm mostly thinking about a situation where I have the results of a cross-validation that someone else did and want to see if my model can beat their average of 80% accuracy. In that case, I would not have their exact folds or perhaps not even how many folds they used, so a paired t-test would not be possible. What would be the pitfalls of comparing to their metric or to their 5 metrics on the 5 folds when I don't know exactly how they allocated the observations into folds?

$\endgroup$
1
$\begingroup$

There seem to be a few elements of confusion here. I'll try to answer clearly, but would definitely appreciate input from others.

1. What is cross-validation for?

Cross-validation is generally used as a mechanism for figuring out which of several models might be a better fit for data based on a training set before choosing a model to apply to the test set. This is commonly done when tuning hyperparameters (such as penalty factors in regularized regressions), and in other scenarios as well.

In such a case you would want to use the same folds for cross-validating each model, because you want to identify differences in performance attributable to the models themselves, and not due to effects of randomly partitioning observations into different folds. Importantly, cross-validation is part of the model training process, not final model evaluation.

Cross-validation is not for finding the "best" performing model overall. Model performance is based on application to a held-out test set of observations. The idea is that the model performance will often be artificially high in the training data due to idiosyncrasies among the training data set which are not present in observations overall, and application to the test data suggests how the model might generalize to totally novel observations. Even if cross-validation results are discussed in a study, they are not the appropriate value to compare a model prepared independently and trained on different data.

2. During cross-validation, how would one assess which putative model is "better"?

This question can be a bit controversial, namely in that accuracy (such as of classification) is not a very strong method of identifying the best model during cross-validation. It will work with varying degrees of quality depending on what you're modelling. But better measures are varieties of loss functions, which could be mean squared error, binomial deviance, or many others (it's very situation dependent).

How you would compare the output of those loss functions is usually fairly straightforward-- you want minimum error, minimum deviance, etc. Performing a paired t-test isn't really appropriate. Such a test would tell you something about the distributions underlying your variables for data points that are connected in time (like measures of some variable before and after a treatment is applied) in the context of a specific null hypothesis and alternative hypothesis set. That sort of paired relationship doesn't apply to observations subjected to different models one-at-a-time, and in a mutually exclusive way.

3. What should be done instead?

As outlined above, model performance is assessed on novel data rather than training data. In model development that will be the test set, but it can just as well be application of the model to totally new data. If the models to be compared are similar in complexity, a proper loss function is still ideal. But in many situations classification accuracy is necessary (if that's what your boss is asking for, that may be what you have to provide).

In that case simpler counts of correct classification (9 out of 15 were correctly classified) are appropriate. Better still are measures of model sensitivity and specificity (the model's ability to identify true positives and true negatives, respectively, as in binomial classification) and positive/negative predictive value (how reliable any given classification suggested by the model is) can be very effective.


tl;dr: Cross-validation is the wrong stage to be comparing an already-trained model to a different model that is being trained, and t-tests are not informative in comparing cross-validations. Loss functions or classification accuracy are more appropriate (to varying degrees in varying situations) and should be based on data not used in training.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Let's say I'm doing MNIST. There are 60,000 images designated as training data, 10,000 designated as testing data. As I tune the hyperparameters of my model (likely convolutional neural net, so filter size, filter number, maxpooling size, etc), I would work on the 60,000, partitioning that into ten groups of 6,000 (600 per number) for my ten-fold CV. Once I have my hyperparameters optimized, check it out on the 10,000 designated as test data. Correct? $\endgroup$ – Dave May 19 '19 at 17:10
  • $\begingroup$ @Dave Convolutional neural nets are not my expertise, but I presume it works similarly to other techniques (you may want to explore that assumption further). You are correct about tuning the hyperparameters through k-fold CV on your training set. You would then apply those to the test set, as you describe. The performance on that test set is a reasonable result to compare against other putative models. Once you have selected a model, it is common to re-train on the entire data set (using that model and its hyperparameters) to make maximum use of the information. $\endgroup$ – Upper_Case May 20 '19 at 1:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.