- What is the actual goal here? I get that we take an average of the k rounds of cross-validation that we do, but what is our output? I've read that we discard the result of each round after calculating the validation error, so how do we actually produce a model from this?
The goal is to obtain an accurate estimate of the model's performance. Cross-validation produces unbiased model predictions for every example in the dataset. We use these predictions to compute metrics, like accuracy or log loss, for the training dataset. When we are actually building a model, we will use the entire dataset rather than cross-validation.
- What is the difference between a validation and test set, if any? i.e., we normally separate our data into training and test data, but do we then further divide our training data into validation sets, and keep our test data aside during the cross-validation process, or do we run cross-validation on the entire data set?
A validation set is a set of labeled examples used during the training phase, but not actually for training the model. This can be done in many ways. For example, when building a model through an iterative process (like gradient descent for neural networks), we can use a validation set to decide when to stop iterating. The performance on the training set almost always improves after each training step, but will eventually overfit and produce poor predictions on unseen data. In this scenario, we can test the model on the validation set to detect when overfitting occurs.
A test set, on the other hand, is only used to evaluate the model after training is complete. Optimising our training for performance on the validation set means we are slightly biased towards it, and we should finally test on a truly held-out set of labeled data to evaluate the performance of the model.
Cross-validation is intended to "replace" the need for a test set for providing an unbiased estimate of model performance, so you usually would not bother with having a test set when doing cross-validation. However, if you have a large dataset it is generally recommended to split into train/test rather than to use cross-validation.
