Given the randomization, it is unlikely that there will be a dramatic change from one run into the next one in the loop of the cross-validation.
This assumption is wrong, it's true only if the training data is large and representative enough with respect to the features, the target class and the complexity of the model. The test set must also be large enough. In other words, if the model overfits, i.e. captures patterns which happen by chance in the current training set, the performance is likely to differ across the runs. Of course this can also happen if the current test set is small (for instance doing 10-fold CV with 100 instances means that the test set for a run contains only 10 instances).
Given the performances inside the cross-validation, what question does the cross validation answer ? Assuming that the performance is low, does it mean that we should abandon training the model ?
Cross-validation provides a statistically more reliable estimation of the performance than a single training/test set split. A simple way to observe this is to train many different models where some hyper-parameters vary. It's likely that the best combination of parameters according to a single training/test set split would be different from the best obtained by averaging across the k-fold CV runs. This is because with a sufficiently high number of models, it becomes very likely that one of them happens to perform better on a particular test set by chance. Averaging across several runs obtained on different test sets reduces this risk.
Beyond performance evaluation, CV also helps diagnose other problems which are not visible with a single training/test set split thanks to the variations across runs.
"performance is low" is interpretative, it depends on the context. Anyway the decision to abandon a project or not usually depends on many factors. But in some cases one might want to reconsider the design and/or method if performance is not satisfactory, this is not specific to CV.
I could understand the aim of the k-fold cross validation, if training the whole model is expensive in time or other resources. In that scenario splitting the training set in k subsets on which the validation is done, could potentially provide a forecast for the performance of the model. Given that even on big data, fitting the model on the entire training set is done with acceptable time complexity, I thus do not understand the role of cross-validation.
This reasoning seems illogical to me, given that $k$-fold CV:
- is only meant to provide an estimation of performance, not to train the final model.
- requires $k$ times more resources than a single train/test split.
It's the opposite: although it's more resource-intensive, CV is sometimes worth running for the advantages explained above. It's not a necessary step though.
In case we use many estimators on the same data set aiming to pick the best one, eg. SVM, logistic regression, deep learning, do one has to perform the k-fold cross-validation on each one of them ? Can one expect that the performance of the k-fold cross-validation will be similar along the different estimators ? What is the best strategy ?
This is an interesting point and a frequent source of confusion. First, no, the performance of the different models is not necessarily similar.
- Any form of selecting an optimal model based on performance, e.g. varying hyper-parameters, subsets of features or learning algorithm, is by itself a training stage. This implies that the performance obtained by the best model is not (necessarily) its true performance, so it must be evaluated on a fresh test set.
- Technically this doesn't mean that CV must be used, it just means that the data must be split three way:
- The training set is used to train all the models
- The validation set is used to evaluate all the models
- The best model is selected and only this model is evaluated on the final test set.
- But very often one uses CV instead of training+validation in the first step, this way the performance of all the models is evaluated more reliably. This has the additional advantage that CV takes care of the splitting, so in this case:
- Use CV to evaluate all the models on the training set.
- The best model is selected and only this model is evaluated on the test set.