Validity of cross-validation for model performance estimation

Question

When applying cross-validation for estimating the performance of a predictive model, the reported performance is usually the average performance over all the validation folds. As during this procedure, several models are created, one model has to be chosen as the model which is actually used for prediction on real-world samples (e.g. in a product).

I am curious whether it is really valid to report the validation performance as estimated performance of the final (selected) model (as the performance was derived using all the other models which were created during the validation procedure, but are not considered when using the final model for predictions).

I would expect that the deviation of the selected model's performance might vary drastically from the average performance of all models (depending on several factors such as the used algorithm and validation scheme).

Why is cross-validation used to estimate the performance of a predictive model despite the given fact (e.g. in many peer-reviewed scientific publications)? Wouldn't it always be better to conduct an additional performance evaluation with the selected model on an independent test set and report the resulting performance alongside the validation performance?

Answer 1

Cross-validation is used to estimate the performance of a certain type of model on a specific dataset.

one model has to be chosen as the model which is actually used for prediction on real-world samples (e.g. in a product).

Selecting one of the models obtained during cross-validation is not appropriate, and proceeding in this way would indeed cause the problem that you mention. The correct methodology is to train the final model on the full training data after cross-validation (i.e. independently from the models trained during CV). This way the performance obtained through CV is representative of the expected performance of the final model.