I have been working on a pretty simple text classifying module (tfidf + Random Forest). My manager insisted on using a simple .7/.3 split rather than doing cross validation, then was adamant about putting the model trained on 70% of the data in production (rather than the model trained on the whole thing). Her rationale was that the model would be more "predictable" and that anything we would gain from adding the remaining 30% would be negligible.
Out of curiosity, I ran a few test using cross-validation. First with 3 folds, then incrementally up to 10. With every increase in fold, the model's precision increases (first quite strongly: there's a 3 point increase in precision from 3 folds to 5 folds, the gains then become more marginal with each fold but still add up to a 5 point boost when reaching 10 folds). The variance between the cross-validation scores is always very low.
Am I right to assume that this can be interpreted as meaning the model is very likely to benefit from being trained on the whole dataset rather than on just 70%?
I understand there may be a risk of overfitting on the training data but a) I don't really see how the risk would be significantly mitigated by training on 70% of the data only, b) these are quite formal, standardized communications and unseen data is unlikely to differ significantly from the training datasets.
