5
$\begingroup$

I have been working on a pretty simple text classifying module (tfidf + Random Forest). My manager insisted on using a simple .7/.3 split rather than doing cross validation, then was adamant about putting the model trained on 70% of the data in production (rather than the model trained on the whole thing). Her rationale was that the model would be more "predictable" and that anything we would gain from adding the remaining 30% would be negligible.

Out of curiosity, I ran a few test using cross-validation. First with 3 folds, then incrementally up to 10. With every increase in fold, the model's precision increases (first quite strongly: there's a 3 point increase in precision from 3 folds to 5 folds, the gains then become more marginal with each fold but still add up to a 5 point boost when reaching 10 folds). The variance between the cross-validation scores is always very low.

Am I right to assume that this can be interpreted as meaning the model is very likely to benefit from being trained on the whole dataset rather than on just 70%?

I understand there may be a risk of overfitting on the training data but a) I don't really see how the risk would be significantly mitigated by training on 70% of the data only, b) these are quite formal, standardized communications and unseen data is unlikely to differ significantly from the training datasets.

$\endgroup$
1
$\begingroup$

It may or may not indicate that more data will help.

The correct way to see if model performance increases with more data is to create a learning curve. Train your model on a small subset of your data with cross-validation, and then keep increasing the amount of data used. This is not just reducing/increasing the number of folds. It is actually limiting yourself to a small amount of data for your 10-fold cross-validation and increasing it, evaluating the performance at each increase. Here's more about a learning curve.

$\endgroup$
0
$\begingroup$

No. Because, as you mentioned, it could lead to overfitting. Another concern should be re-sampling error. It's possible that the folds you chose and the data that was chosen for that training cycle has generated completely different results than another cycle would.

$\endgroup$
  • $\begingroup$ Do you have a reason/reference to believe that this can overfit? $\endgroup$ – Ben Reiniger Mar 19 at 2:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.