4
$\begingroup$

I currently have a model that has a pretty large dataset (50ishMB) and was performing pretty well with a 80:20 split. However, when I tried changing it up to a 50:50 split, the model performed 28% better than the 80:20 split. Note this is a time-series problem.

Since I have evaluated the model to perform better in that data split, is it fine to more forward with it or should I continue with the 80:20 split? Why?

$\endgroup$
5
  • 2
    $\begingroup$ Try repeating a few times w different 80/20 splits. See much variance? Maybe you got an unlucky split the first time… $\endgroup$
    – G__
    Dec 5 '21 at 17:43
  • $\begingroup$ It's a time-series problem so not a lot will change. $\endgroup$ Dec 6 '21 at 13:49
  • 1
    $\begingroup$ Dont think the data type matters. You’ve already reported a substantial (28%) swing by changing the split. Above is a suggestion to help understand this. $\endgroup$
    – G__
    Dec 8 '21 at 0:31
  • $\begingroup$ 50MB is not large, it’s tiny. And there are two different “splits”: The first is your train/validation split (which might as well be 99%/1% + 100-fold cross validation since your dataset is so tiny). The other split is how many points you’re predicting on each eval (e.g. at inference the model sees 80 points and predicts the next 20 in the time series) which is pretty arbitrary and totally depends on your application. I think you’re confusing the two ratios. Post the code you used to split the data set and each time series in it. $\endgroup$
    – Navin
    Dec 23 '21 at 18:42
  • $\begingroup$ All code for the model is here: github.com/Khosraw/… The function for the split is commented there for you to see. @Navin $\endgroup$ Dec 26 '21 at 3:50
4
$\begingroup$

Assigning more examples for training allows your model to be exposed to more types of hidden patterns in your data & presumably learn a better representation. While assigning more examples for testing gives you a better fidelity of the evaluation over these data patterns. Ultimately this is a project specific tradeoff you make.

CONSIDERATIONS

  • Does the evaluation need to be precise for a specific trained model or could it be a range approximation? If the latter then cross validation let’s you best use all the data to learn & evaluate.

  • Is the model used classical or deep learned? The latter techniques often require way more data so I would favor a 99:1 train/test split over the typical 70:30 or 80:20 splits.

  • Is the problem domain simple or extremely difficult? ie: are there sub patterns, many edge cases, lots of classes to learn with high human disagreement? You can plot learning curves to see how well your model learns with different volumes of data. Once it plateaus you can allocate excess data for evaluation.

  • How expensive is it to get quality & representative labeled data? If cheap consider creating ship-gates (unit test equivalent) datasets to capture various data scenarios.

  • Is the data heavily imbalanced, very noisy or very repetitive? if so I would apply smarter splitting techniques like stratified sampling, imbalance handling, etc over just random splitting.


It is common to allocate more data for training. I would attribute the perceived improved performance in your 50:50 data split simply to producing a less over-fitted model due to having fewer learning examples.

$\endgroup$
4
$\begingroup$

Using a 50:50 split is usually not recommended. People usually keep more data for training and less data for testing/validation.

The more train data you have, the better the model captures different scenarios of data and the more test data you have, the better your trained model will be evaluated. So it's a trade off between the two, and ultimately you have to decide which do you prefer more.

Since you mention you have a large dataset, choosing 50:50 wouldn't be that much of a problem compared to if you has a small dataset. But there would still be some data patterns you would be missing which in turn will make your model less generalizable so keep that in mind!

A possible solution to this trade off is cross-validation (preferably nested cross-validation). That way even if you have less training data, your model will use all of that data in the best possible way.

Cheers!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.