I am a beginner in the field of machine learning, I have small doubts for which I did not find any suitable answer. How to select the best model for prediction of unknown data. I have learned two ways I am not able to figure out which one is correct one.
Stating with, train your model by splitting the data into training and test data, then fit the model to predict the test output and error, shuffle the data and average the error over some 100 or more cycles. This will give us an average error (rmse for test) over 100 cycles (python way random state). Now to predict the unknown data (validate the model), which model should I consider the model for prediction.
1-Which have rmse closer to average rmse: pick a model which reports error approximately equal to the average error achieved over 100 cycles to predict the unknown data and called that as the model for prediction, or
2-Best performance: Chose a model out of 100 which is best (lowest rmse for test) for prediction of unknown data?
Another thing I am also struggling is that if considered the lowest error model as the model for unknown data prediction, by keeping X_train and y_train same of the best model.And If I chose my X_test and y_test from the same database (10% only) over 400 different times and predict the error, will it be an overfitted prediction?
Thanks in advance