0
$\begingroup$

I have 10000 customer data of a supermarket. And I want to split the data into training set and testing set. So,which train test split gives me a better accuracy?

$\endgroup$
1
$\begingroup$

It sounds like you have a lot of data, so probably a simple train-test split is enough. No need for cross validation.

I would just use something like 75-25. That is in fact the default value in sklearn.

I would use less data in the training only if your algorithm is too slow and cannot cope with the extra data. In that case, instead of throwing away the data, you might as well use it for validation or for testing hyperparameters.

All this being said, more important than all is how you split the data. You should make sure that customers in the testing data are not in the training data to make sure your algorithm is generalizing, and not merely memorizing customers. This is standard procedure in medical data mining, and it is very important. Make sure you do not have customer overlap.

You might also want to make sure that the distribution of the variable you want to predict is similar in the training and the testing data.

$\endgroup$
0
$\begingroup$

I will assume that the dataset here is being split into training and validation sets.

When you split up your dataset into training and validation sets, you need to take care of that you don't throw away too much data for validation because it is generally seen that with more training data, we get better results in most of the cases. So 50:50 is too bad, 60:40 is fine but not that good. You can make it 80:20 or 75:25 for getting better results.

$\endgroup$
0
$\begingroup$

In a draft copy currently being written by Andrew Ng, he discusses about the amount of data in train-test dataset. My understanding from the book, The traditional and most common value is 70-30 or 75-25. If you have 10k or 30k samples, it is fine to go with 70-30 split. But when dealing with Big-data, for example if you have 1 million samples, it is not recommended to have 30k samples as test data, so in that case, 90-10 is actually okay. Because 10k test samples can pretty much provide an intuition about the model.

in brief: for less samples, go with recommended 70-30 split, for much higher samples, go with number of samples

Draft copy link : ML Yearning

$\endgroup$
0
$\begingroup$

Any train-test split which has more data in the training set will most likely give you better accuracy as calculated on that test set. So the direct answer to your question is 60:40. And 99:1 would even give better accuracy...

However, at the end of the day, you are not interested in the accuracy on your test set. You are interested in the "real" accuracy, which gets estimated by your test set. And you better want to make sure that the test set can predict that accuracy well.

So, which split should you pick?

  • Make sure "enough" data is in the test set. What "enough" means depends on your dataset (the number of classes, number of features)
  • If you rather want a good estimate of your real error, make the test set bigger
  • If you doubt that you can get more training data and you think the more data in your training set will improve your model (in real, not on the test set) a lot, then "sacrifice" a bit for the training set.

Whatever you do, make sure you define your training set before you start your experiments.

$\endgroup$
0
$\begingroup$

I read all answers, I think the simplest answer to this question is based on the understanding of the train - test strategy. There is NO exact or correct answer to this question. Any split that can guarantee I am not under fitting or over fitting the data is a good a split. under fitting and over fitting are two different problems and are directly connected to the bias error and the variance error. You are highly recommended to read these two tutorials: http://scott.fortmann-roe.com/docs/BiasVariance.html http://scott.fortmann-roe.com/docs/MeasuringError.html

$\endgroup$
0
$\begingroup$

If you have enough data, then you can actually go for a 50-50 split but there is no such thing as what would be better, depends completely on the amount of data you have and the complexity of the task you are trying to perform.If you train it on enough data, the size of the test set is of no concern. The whole reason for splits comes from the fact that we often have limited and finite data and we want to make the best use of it and train on as much data as we can. So go by the heuristic, and do a 75-25 split. But don't forget to cross-validate on the training set, I would recommend a stratified-K-fold. If your performance metric is suffering, the split would be the last reason for it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.