How to correctly approach the generation of a training/test set? I am doing several experiments testing the generalization ability of my neural network model, so my test set is different from my training set in all experiments (for example, in one experiment the structure of sentences is the same between the training set and the test set, but in the training set I use one set of words and in the test set I use another set of words). So my question is: to be able to compare accuracy between experiments, do I have to maintain a similar size of training set/test set between the experiments? Should I only make sure that the size of the training set is always similar or the size of the test set? In one experiment I have a dataset of size 29160 that I generate for training and in other experiments I have bigger datasets that I generate for training (sometimes of size 122472), so should I use the whole dataset in that first experiment and draw a sample of around 30000 examples from other bigger datasets in other experiments to use as a training set?
2 Answers
There is no fixed rule while selecting the size of the training set and testing set. Its all about trial and error, so try out different ratios 80-20, 70-30, 65-35 and pick one that gives the best performance result.
Its suggested in several machine learning research articles to generally opt for
- Training dataset to be 70% (for setting model parameters)
- Validation dataset to be 15% (helps to tune hyperparameters)
- Testing dataset to be 15% (helps to access model performance)
If you plan to keep only split data into two, ideally it would be
- Training dataset to be 75%
- Testing dataset to be 25%
In case of extremely large datasets which typically can go to millions of records, a train/validation/test split of 98/1/1 would suffice since even 1% is a huge amount of data.
UPDATE:
Do ensure you split your training, test datasets using an algorithm and not manually.
One major concern while splitting data is to ensure the data is not imbalanced. For instance, you have 5 classes that needs to be classified in your ML problem. You need to ensure the train and test dataset has sufficient data with these 5 classes for model to give best performance. If you manually split the data chances are your dataset might have only 10% of data in varied classes or worse it might only have 3 classes in the test/train data. To ensure data is not imbalanced use stratify in sklearn's train_test_split
X_train, X_test,y_train,y_test = train_test_split(X, y, test_size=0.15, train_size=0.15,stratify=X['YOUR_COLUMN_LABEL'])
-
$\begingroup$ Maybe I didn't explain it properly, but I generate two separate datasets to use for training and testing, but they are quite big in most of my experiments, so I draw samples with like half of the total number of examples from both datasets and use one as a training set and the other as a test set $\endgroup$– lmtr339Commented Jul 9, 2021 at 17:22
-
$\begingroup$ Have you manually chosen the train & test datasets? $\endgroup$ Commented Jul 10, 2021 at 19:20
-
$\begingroup$ I generate two different datasets and I use one for training and another for testing $\endgroup$– lmtr339Commented Jul 12, 2021 at 8:16
-
$\begingroup$ As mentioned in my answer, just ensure that your training/testing data is not imbalanced.
stratifyin sklearn helps to achieve this. Choose a method that's feasible to your scenario $\endgroup$ Commented Jul 21, 2021 at 10:03
Size of the training set and test test should not be in similar size why ? because what ever the model your are testing it will fist applicable on the training set equal size of data will create noise.
In the machine learning world, data scientists are often told to train a supervised model on a large training dataset and test it on a smaller amount of data. The reason why training dataset is always chosen larger than the test one is that somebody says that the larger the data used for training, the better the model learns.
Larger test datasets ensure a more accurate calculation of model performance. Training on smaller datasets can be done by sampling techniques such as stratified sampling. It will speed up your training (because you use less data) and make your results more reliable.
