1
$\begingroup$

I am using Matlab to train a feedforward NN using Cross validation (CV) approach. My understanding of CV approach is the following. (Please correct me where wrong)

  1. Let X be the entire dataset with Y as the label set. Split X into 90/10 ratio to get: [Xtrain,Xtest] using holdout approach by calling the cvpartition(Y,'Holdout',0.1,'Stratify',true)

  2. Apply CV on Xtrain: For every fold I calculated the accuracy. At the end of the CV loop I have an average accuracy score. Let accCV denote this variable. Inside the CV loop xtrain is further split into [xtrain_cv,xtrain_val].

  3. After CV loop, I reinitialze the weights and re-train a new model using Xtrain. Then I get a training accuracy which I denote by the variable accTrain.

  4. Using the model obtained in Step3 I test for evaluating the model's purpose and consider this to be the generalization performance that is the performance when we have an unseen future data, Xtest. I call this accuracy as accTest.

Question1: Is it possible that accCV will be less than the accuracy over the train set Xtrain when not using the CV approach? That is I call the NN model over Xtrain only once and record the accuracy and denote it by variable accTrain, then is it possible that accCV ~ accTrain?. Or intuitively, accCV should be close to the accuracy when not using CV approach since the dataset is the same which is Xtrain. If this is the case, then why use CV when outside the CV we do not reuse the model that was created inside the CV? What does it tell us?

Question2: If accCV < accTest but the accuracy on the entire dataset Xtrain without using CV is close to that of accTest (accTrain ~ accTest) are we doing something wrong? What is the best case scenario? Is it accCV ~ accTest?

$\endgroup$
2
$\begingroup$

It is expected that accCV < accTrain: the former is the accuracy on the test folds (averaged over all the splits), so represents models' scores on unseen-to-them data. Similarly, you would expect accTrain > accTest.

There are two main reasons to evaluate a model, whether by k-fold cross-validation or simple train/test split: for hyperparameter optimization / model selection, or to estimate future performance. (N.B., k-fold should generally give better predictions than simple train/test split.) If you make any decisions based on the scores, then they no longer represent unbiased estimates of future performance. If, in your setup, you make no decisions based on step 2, then you should expect accCV ~ accTest, and there's no real reason to include that step. If you do make decisions based on step 2, you may expect accCV > accTest, though the gap is probably substantially smaller than the gap in accTrain > accTest. You may see discrepancies here, due to natural variation in the datasets, or perhaps data leakage.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for your asnwer, however few points are unclear. Could you please clarify?(1) I could not understand what you meant:"If you make any decisions based on the scores, then they no longer represent unbiased estimates of future performance." Did you mean to have a separate unseen data representing future data that should not be used in the CV loop and use this dataset for performance evaluation?(2)If accCV~accTest then is it a good or bad thing?(3) Is my creation of the test set (future unseen) which is never used in the CV loop correct? I called accTest for this test set. $\endgroup$ – Sm1 Jan 27 at 19:56
  • $\begingroup$ (1,3) Your setup, with a k-fold CV on the train set and a separate test set, is most commonly used when you want to do model selection: you fit many models with CV on the train set, choose the model pipeline with best accCV, refit that pipeline to the entire training set, and finally score on the test set getting accTest. If you are not doing model selection, then you can get rid of either your step 2 or 4. Read some of the Related questions. (2) accCV~accTest is good; if not, then your test set is not iid with train set, or you have some data leakage, or... $\endgroup$ – Ben Reiniger Jan 27 at 20:17
  • $\begingroup$ Just to confirm, by accTrain you meant the training accuracy on 90% of the split data set that was used inside the CV procedure and which is not the train data in the CV fold. Also, if we do training by CV then there is no need for a separate test set for generalization purpose since the test fold will serve that purpose. Is my understanding correct? Thank you very much for your help andclarifications. $\endgroup$ – Sm1 Jan 28 at 18:55
  • $\begingroup$ Yes on both counts. $\endgroup$ – Ben Reiniger Jan 28 at 23:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.