Best model for Machine Learning

Question

I am a beginner in the field of machine learning, I have small doubts for which I did not find any suitable answer. How to select the best model for prediction of unknown data. I have learned two ways I am not able to figure out which one is correct one.

Stating with, train your model by splitting the data into training and test data, then fit the model to predict the test output and error, shuffle the data and average the error over some 100 or more cycles. This will give us an average error (rmse for test) over 100 cycles (python way random state). Now to predict the unknown data (validate the model), which model should I consider the model for prediction.

1-Which have rmse closer to average rmse: pick a model which reports error approximately equal to the average error achieved over 100 cycles to predict the unknown data and called that as the model for prediction, or

2-Best performance: Chose a model out of 100 which is best (lowest rmse for test) for prediction of unknown data?

Another thing I am also struggling is that if considered the lowest error model as the model for unknown data prediction, by keeping X_train and y_train same of the best model.And If I chose my X_test and y_test from the same database (10% only) over 400 different times and predict the error, will it be an overfitted prediction?

Thanks in advance

Answer 1

If you use the test set for model selection (it does not matter whether you use some random state, or do hyperparameter tuning) then it is dirty and can no longer be used to predict the prediction performance.

The general workflow you should follow is this:

1. Split into train and test. Lock "test" in a safe.
2. Use cross-validation on the train set only to tune parameters.
3. Fix the hyperparameter to the best setting found. Write them in boldface as 'final parameters' into a stone plate on your wall.
4. Train a new classifier on the entire training set using exactly these parameters
5. Get the test set from the safe, and run the classifier on the test set exactly once to get a prediction of how well your classifier is going to work in the future
6. If the results are substantially worse than those from 2.: panic. It's not going to work.
7. Optional: If you are confident that hyperparameters do not depend on the data size but that you need every little bit of training data, you could train a classifier on train+test with the in-stone parameters. This is risky though, a you don't have a test set anymore.
8. Deploy, but monitor that it really performs as good as predicted.

The golden rule is: if you used the test set, you must not change the model anymore. You've got one shot to score.

That is why you split the train set again in CV.