I am a beginner in the field of machine learning, I have small doubts for which I did not find any suitable answer. How to select the best model for prediction of unknown data. I have learned two ways I am not able to figure out which one is correct one.

Stating with, train your model by splitting the data into training and test data, then fit the model to predict the test output and error, shuffle the data and average the error over some 100 or more cycles. This will give us an average error (rmse for test) over 100 cycles (python way random state). Now to predict the unknown data (validate the model), which model should I consider the model for prediction.

1-Which have rmse closer to average rmse: pick a model which reports error approximately equal to the average error achieved over 100 cycles to predict the unknown data and called that as the model for prediction, or

2-Best performance: Chose a model out of 100 which is best (lowest rmse for test) for prediction of unknown data?

Another thing I am also struggling is that if considered the lowest error model as the model for unknown data prediction, by keeping X_train and y_train same of the best model.And If I chose my X_test and y_test from the same database (10% only) over 400 different times and predict the error, will it be an overfitted prediction?

Thanks in advance


1 Answer 1


If you use the test set for model selection (it does not matter whether you use some random state, or do hyperparameter tuning) then it is dirty and can no longer be used to predict the prediction performance.

The general workflow you should follow is this:

  1. Split into train and test. Lock "test" in a safe.
  2. Use cross-validation on the train set only to tune parameters.
  3. Fix the hyperparameter to the best setting found. Write them in boldface as 'final parameters' into a stone plate on your wall.
  4. Train a new classifier on the entire training set using exactly these parameters
  5. Get the test set from the safe, and run the classifier on the test set exactly once to get a prediction of how well your classifier is going to work in the future
  6. If the results are substantially worse than those from 2.: panic. It's not going to work.
  7. Optional: If you are confident that hyperparameters do not depend on the data size but that you need every little bit of training data, you could train a classifier on train+test with the in-stone parameters. This is risky though, a you don't have a test set anymore.
  8. Deploy, but monitor that it really performs as good as predicted.

The golden rule is: if you used the test set, you must not change the model anymore. You've got one shot to score.

That is why you split the train set again in CV.

  • $\begingroup$ 3rd point is cool $\endgroup$
    – Aditya
    Commented Apr 14, 2018 at 11:41
  • $\begingroup$ @Anony-Mousse But I did not understand the role of data shuffling, Is it not required? Points I used to run for loop on data shuffling and then tune hyperparameter, and finally check for rmse over 100 different run. Then either I can choose the model with best rmse or average all rmse values. $\endgroup$ Commented Apr 15, 2018 at 7:34
  • $\begingroup$ For many algorithms order does not matter. But cross-validation would usually involve shuffling. $\endgroup$ Commented Apr 15, 2018 at 7:38
  • $\begingroup$ @Anony-Mousse Suppose I have 90 Data in total the procedure would be (for regression) 1-Split the data in 90-10 ration so 81 as training and 9 Test, and Keep the test safe 2-For my case apply KRR with GridsearchCV and CV=5 for hyperparameter tuning 3-Model with the best hyperparameter, select it. 4-Finally predict it on the test data. Is it correct, what I understood? $\endgroup$ Commented Apr 15, 2018 at 11:10
  • $\begingroup$ Essentially yes. Except your data is so small that the error margin of your accuracy estimate will be huge. You really should get more data. $\endgroup$ Commented Apr 15, 2018 at 17:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.