I read in many different sources that I need to split my data into a training set and a test set. Then I have to make sure that the algorithm is trained only on the training data, and do my best to keep the test data from leaking into the training process.
To avoid learning insignificant details of the data (which will increase the algorithm's generalization ability) I can further split my training data into the proper training set and a validation set multiple times, and choose the parameters of the algorithm that give the best average performance for all such splittings.
Finally I evaluate my algorithm on the test set, and get some numbers: MSE, RMSE, etc. But do these numbers really show, how good is my algorithm, and aren't they affected by the test set?
Of course, I didn't use my test data during training but the parameters of the algorithm that I got are valid only for this particular splitting into the training and test sets. If I split my data differently, I will get different MSE, RMSE, and different optimal parameters.
I can then do many different splittings into training/test, and calculate the error for each split. I that case how is it different from the cross-validation that I did during the first split? Do I need to further divide my training data into the proper data set and a validation set?
If I do many different splittings into the training an test sets, doesn't it mean that my test data leak into the training process because some data were in the test set for one splitting but in the training set for another splitting?