I am a beginner in machine learning, and I hope someone can help me.
In Python's 'scikit-learn' library, the function 'train_test_split' splits the dataset into training and test sets. This is done in a random way (possibly using a seed to obtain the same result in repeated executions). But, with a single random split, how much can we trust the result (classification accuracy) obtained through the 'fit' method? I mean... if we are "unlucky", we may obtain a very bad outcome (or the opposite, if we are "lucky"). Shouldn't we repeat the split/fit procedures multiple times (e.g., 100) and then average the classification accuracies obtained? (possibly after parameter tuning by means of cross validation).
I am asking this because previously I used the Python's libraries of Orange Data Mining, which include a method ('proportion_test') that splits the dataset into training and test sets and then evaluates it according to a specific classifier, repeating the operation a specified number of times (e.g., it performs 100 iterations of 70:30 test). My question is: should I manually do this also with the split/fit functions in scikit-learn? (e.g., 100 iterations using 100 different random seeds). Would results be better?
I am very confused about this...
I want to stress that I know about cross validation, leave one out, etc. But, if I understand it right, these techniques are used for model validation (i.e., model parameter tuning). My question is whether the final evaluation of the model should be based on repeated split/fit operations. For example, in the book 'Introduction to Machine Learning with Python' (by Andreas C. Müller and Sarah Guido, O'Reilly), the suggested operation pipeline is: (1) split the original dataset into training set and test set; (2) perform parameter tuning (i.e., best parameter selection) using cross validation on the training set; (3) re-train using the just found best parameters with the training set; (4) perform the final evaluation (calculating the classification accuracy) using the trained model and the (single) test set. My question is: is this enough? Or, once the classifier has been trained using the best parameters (step 3 above), should I repeat the split/fit procedures multiple times on the whole dataset (original training + test sets) to obtain more reliable results? I did this using the Orange library, but maybe it is not necessary (it is not done in the book quoted above).
THANK YOU VERY MUCH in advance for your help!