0
$\begingroup$

If I am predicting probability of a business to reach (x) milestone (classification 1), but the only data I have is live production data, how much of the production data should I use to train the model? My assumption is that if I use all the data, the probability of any business that hasn't reached the milestone already (classification of 0) will most likely stay 0... as I just trained the model that it should be 0.

As a caveat, I know that it is commonplace to use something like an 80/20 or 70/30 split for training/test sets -- most of my fruitless searches have brought up that answer, but my question is if I should take 10% of my production data to then split 80/20 or 70/30 between training and test, as to not overfit the model.

My dataset is 30k records, so my first inclination was to use 3-5k records out of that for training/test.

$\endgroup$

2 Answers 2

1
$\begingroup$

You can use 100% of the data. The more data you feed to the model, the more accurate the prediction is going to be. If you have only two features - time and live production data, 30k records is not going to be a problem. Select a timepoint - somewhere between 70%-95% of the time passed, and use the data before the time point as training data and the data after the time point as test data. You want to use the newest data for testing. You can try using different time points to see how that will affect the model.

$\endgroup$
0
$\begingroup$

Note that your dataset might be considered time-series. In time-series it is common to evaluate a model via back-testing rather than a single train/test split. Back-testing uses as training data all data before time T, and as test data all data after time T, and then we do this for many different values of T and average the test performances for each T.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.