Machine Learning algorithms and Cross Validation, the best practice

Question

I'm new in Machine Learning, and I'm studying the main concepts behind algorithms from the mathematical point of view. I'm also trying to start implementing some algorithms for regression purposes using sklearn library.

I would like to summarize the main steps in ML and provide the question:

1. I understand the dataset, clean and prepare, identifying a set of suitable algorithms for such dataset.
2. Train the algorithms and see the performances, choosing the best one.
3. Hyperparameter tuning of the best performing algorithm using Random or Grid Search with cross-validation between 5-10
4. I have the best parameters for training the algorithm, I train the algorithm with such params.

Now the question that arises for which I am not clear what to do.

May I consider to train the algorithm with the best parameters also performing the cross-validation? or Have I to train the algorithm without the cross-validation?

In short, The model puts into production is the model trained with cross-validation (for instance using KFold for training in step 4) or the model trained without (step 4)?

Answer 1

There are three types of data splits and each have a purpose:

• Training data is used to find model parameters.
• Validation data is used to find the best hyperparameters.
• Test data is used to estimate how well the model will perform on unseen data.

The final model put into production should use the best hyperparameters found through cross validation and be trained on all data available (train, validation, and test).

Answer 2

Cross-validation (CV) is a method meant essentially to accurately evaluate a model using some training data. As a consequence CV doesn't have to be used when training the final model, usually one simply uses the whole training data for that. Note that using one of the CV models would have two disadvantages: there's no reason to select one or the other (the performance on one split is not relevant), and it doesn't exploit the whole training set.

I also notice that you didn't mention any test set evaluation in your steps. A common mistake is to select a model based on parameter tuning (step 3) using CV, and just assume that the CV performance is reliable. It is not because the fact that a choice was made (whether about parameters, models, features...) means that this step is still part of the training process, therefore there needs to be evaluation on a different test set afterwards. In other words: the fact that some parameters perform better than others could be due to chance, so it's important to evaluate the selected model (and only this model) on a fresh test set.