Machine learning - 'train_test_split' function in scikit-learn: should I repeat it several times?

Question

I am a beginner in machine learning, and I hope someone can help me.

In Python's 'scikit-learn' library, the function 'train_test_split' splits the dataset into training and test sets. This is done in a random way (possibly using a seed to obtain the same result in repeated executions). But, with a single random split, how much can we trust the result (classification accuracy) obtained through the 'fit' method? I mean... if we are "unlucky", we may obtain a very bad outcome (or the opposite, if we are "lucky"). Shouldn't we repeat the split/fit procedures multiple times (e.g., 100) and then average the classification accuracies obtained? (possibly after parameter tuning by means of cross validation).

I am asking this because previously I used the Python's libraries of Orange Data Mining, which include a method ('proportion_test') that splits the dataset into training and test sets and then evaluates it according to a specific classifier, repeating the operation a specified number of times (e.g., it performs 100 iterations of 70:30 test). My question is: should I manually do this also with the split/fit functions in scikit-learn? (e.g., 100 iterations using 100 different random seeds). Would results be better?

I am very confused about this...

I want to stress that I know about cross validation, leave one out, etc. But, if I understand it right, these techniques are used for model validation (i.e., model parameter tuning). My question is whether the final evaluation of the model should be based on repeated split/fit operations. For example, in the book 'Introduction to Machine Learning with Python' (by Andreas C. Müller and Sarah Guido, O'Reilly), the suggested operation pipeline is: (1) split the original dataset into training set and test set; (2) perform parameter tuning (i.e., best parameter selection) using cross validation on the training set; (3) re-train using the just found best parameters with the training set; (4) perform the final evaluation (calculating the classification accuracy) using the trained model and the (single) test set. My question is: is this enough? Or, once the classifier has been trained using the best parameters (step 3 above), should I repeat the split/fit procedures multiple times on the whole dataset (original training + test sets) to obtain more reliable results? I did this using the Orange library, but maybe it is not necessary (it is not done in the book quoted above).

THANK YOU VERY MUCH in advance for your help!

Answer 1

You can use KFold cross validation if you want to average the results of the training.

sklearn.model_selection.KFold

This will split your data in a specified number of folds (k) and train your data on all but one folds, then validate on the last fold. This operation is done k times and the results are averaged out.

Normally, how I do is:

1. Train/Test split
2. Model selection + Hyperparameters tuning using KFold on training set
3. Retrain the final model on the whole training set
4. Evaluate on the test set

Note that if you want to check whether your split was 'lucky' or 'unlucky', you can still change the seed, or not give a seed at all and compare the results with different runs.

[EDIT] As stated in the comments below, the seed is controlled by the random_state argument and is mainly there for reproducibility. If you want a different train/test split at each run, just leave the default as is. It's always good to check at least twice to see whether you've been particularly lucky or not but it never happened to me ! :)