How to train and evaluate machine learning models with growing/changing datasets over time

Question

Assume that you have a classification machine learning model, and you start with an initial dataset that contains 3 classes. You split the initial dataset into training/testing spits, you train the initial model and evaluate it, and then overtime, you collect more data for your dataset. Now you have more data that you want to add to you initial training dataset. The question is: How do you you organize you dataset and model training regiment so you can effectively quantify possible improvement between the initial model and the new model?

One possible solutions: if you split the initial training dataset into training/testing data, once you have more data, you just split that data and then add the respective new training/testing splits into the original splits. But this approach feels a little too simplistic, and it could potentially lead to drift in the dataset itself.

This is similar to doing regression testing but for machine learning models, but I am not sure im using the correct technical jargon for it.

Answer 1

The solution you described is definitely an option.

The other motivation for retraining is that the distribution of new data has drifted away from that of the original dataset and that warrants retraining to maintain the same level of model performance.

In such situations, It is possible that you may actually throw away some of your older training data because they may have become stale and include a part of newly available data for training. The remainder new data can then be used for out of sample testing and validation.

You also want to be able to figure out when to trigger the retraining. This is usually done by an independent drift detection process, which keeps track of some metric related to your Business and triggers retraining if it dips below a predefined threshold.

Answer 2

I think an important aspect that needs to be taken into account is whether the new data is more relevant than the older data. If you are collecting data from some physical process that doesn't change, older data is equally valuable as the newer data. Perhaps more commonly the new data is more valuable/relevant because the underlying data generating process changes. For example if you try to predict clicks on ads, people's reaction to ads can change over time and the audience that sees the ad can change as well, so newer data would be better.

If that's the case, you may want to evaluate on newer data only, although simplicity is also a good property. If something simple works, don't discard it because it doesn't make you look smart. ;-)