I have a trained ML Model and new data coming in every week. The new data sometimes vary too much (in a statistical sense) and, therefore, the performance of the ML model degrades. The data-set schema consists of a few numerical columns, few categorical columns, and few text columns.

I have an infrastructure to calculate necessary statistics (mean, variance, skewness, null-values, unique-values) for each feature but am unaware, how much change should I allow between the statistics of the training set and the new data set before I consider retraining the model?

  • $\begingroup$ Maybe you can check the performance of your model on the new data and check that it is consistent with expected performance ? $\endgroup$ – LouisB Nov 19 '20 at 15:44
  • 1
    $\begingroup$ I do not have a way to check the performance of the model. The new data set has no expected output. Currently, it's only by human observation, we detect that our model has degraded and needs to be retrained for the new data. $\endgroup$ – Mathematician Nov 19 '20 at 16:14
  • $\begingroup$ Even if you have no labels in the new data, maybe you can make the assumption that the distribution of labels is constant in time (maybe over a week or something) then you could check that the distribution of predictions in the new data is the same as the distribution of predictions on the old data ? It is not bulletproof but it is a nice check $\endgroup$ – LouisB Nov 19 '20 at 16:39
  • $\begingroup$ That's a very helpful suggestion. But this assumes the label to follow a particular distribution. We thought of the system of comparing with metrics, so the decision of retraining the ML model can be made independent of labels (a generalized approach). $\endgroup$ – Mathematician Nov 19 '20 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.