Is there a link between Training, Test errors based on k fold CV and not doing CV?

Question

I am using Matlab to train a feedforward NN using Cross validation (CV) approach. My understanding of CV approach is the following. (Please correct me where wrong)

1. Let X be the entire dataset with Y as the label set. Split X into 90/10 ratio to get: [Xtrain,Xtest] using holdout approach by calling the cvpartition(Y,'Holdout',0.1,'Stratify',true)

2. Apply CV on Xtrain: For every fold I calculated the accuracy. At the end of the CV loop I have an average accuracy score. Let accCV denote this variable. Inside the CV loop xtrain is further split into [xtrain_cv,xtrain_val].

3. After CV loop, I reinitialze the weights and re-train a new model using Xtrain. Then I get a training accuracy which I denote by the variable accTrain.

4. Using the model obtained in Step3 I test for evaluating the model's purpose and consider this to be the generalization performance that is the performance when we have an unseen future data, Xtest. I call this accuracy as accTest.

Question1: Is it possible that accCV will be less than the accuracy over the train set Xtrain when not using the CV approach? That is I call the NN model over Xtrain only once and record the accuracy and denote it by variable accTrain, then is it possible that accCV ~ accTrain?. Or intuitively, accCV should be close to the accuracy when not using CV approach since the dataset is the same which is Xtrain. If this is the case, then why use CV when outside the CV we do not reuse the model that was created inside the CV? What does it tell us?

Question2: If accCV < accTest but the accuracy on the entire dataset Xtrain without using CV is close to that of accTest (accTrain ~ accTest) are we doing something wrong? What is the best case scenario? Is it accCV ~ accTest?

Answer 1

It is expected that accCV < accTrain: the former is the accuracy on the test folds (averaged over all the splits), so represents models' scores on unseen-to-them data. Similarly, you would expect accTrain > accTest.

There are two main reasons to evaluate a model, whether by k-fold cross-validation or simple train/test split: for hyperparameter optimization / model selection, or to estimate future performance. (N.B., k-fold should generally give better predictions than simple train/test split.) If you make any decisions based on the scores, then they no longer represent unbiased estimates of future performance. If, in your setup, you make no decisions based on step 2, then you should expect accCV ~ accTest, and there's no real reason to include that step. If you do make decisions based on step 2, you may expect accCV > accTest, though the gap is probably substantially smaller than the gap in accTrain > accTest. You may see discrepancies here, due to natural variation in the datasets, or perhaps data leakage.