Cross validation test and train errors

Question

I came across this sort of flowchart:

Below the flowchart, this is what appears:

“Given a training set, cross-validation error is computed for each configuration of tuning parameters (λ,d). The configuration of tuning parameters with the lowest overall cross-validation error is chosen to be the best as it leads to the best model performance. Using the best configuration of tuning parameters, we then train the models M2 and M3 on the original training set and use the original test set to compute the corresponding test RMSEs.”

• They are only mentioning the cross validation error (validation) and never mention the train cross validation error.
• Is the phrase “The configuration of tuning parameters with the lowest overall cross-validation error is chosen to be the best as it leads to the best model performance” correct? I mean, assuming that by “lowest overall cross-validation error leads to the best model performance”, they are referring themselves to the “validation” error of the cross validation technique, I wonder why are they making such assumption? Should we care about the averaged train cross validation error or just the averaged validation error?

I am using a library to play with recommender systems which has a parameter called return_train_measures = True. Then it throws both, train and test errors:

Answer 1

The cross-validation error is calculated using the training set only. Choosing the model that has the lowest cross-validation error is the most likely to be the best model for data that it hasn't yet seen. But it's not necessarily actually the best, so in that sense, strictly speaking, you can say the phrase is wrong.

Even if your sample is a properly randomly drawn from the data generating process, it's still a finite sample. Imagine if the data is one-dimensional: Each example is just a number. The average of the training set could, by sheer bad luck, deviate a lot from the average of the data generating process. It's a simplistic example, but the best model on this data is not necessarily the best on any sample from this process.