8
$\begingroup$

I have customer demographic data that include columns like: age, the first half of the postcode, occupation (there is a defined list of possible occupations), and more. Each month I get a new batch of 1000 rows of this type of data (which is not labelled) and I need to put this into my trained model to predict what item (out of 5 items) each person in the new batch data set is most likely going to buy (a multiclass classification problem).

Each time I receive this data, I compare the summary statistics between the old and new data, and investigate any changes in the distribution of the categorical variables using hypothesis testing. If my tests show that my new batch of data had vastly different summary stats, or distributions to my training set e.g.

The new batch targeted people under 25 only, whereas my training set contains all age groups. The new batch targeted people from a specific area of the UK, whereas my training set contains all possible locations in the UK. Would I need to:

Make any changes to my training set, or my overall workflow, to adjust for this?

As far as I know, this is data drift. Am I correct in saying that?

If the batch data coming in was labelled, so we knew what items these people bought, and there was a sizable difference in the proportion of each product sold, what could I do to quantify this instead of naively adding this new data to the training set and retraining my model?

Thanks

$\endgroup$

1 Answer 1

8
$\begingroup$

As you suggest, that situation could end up your monitoring system indicating a data drift. To evaluate this scenario, let's classify some types of data drift we could have:

  • features drift: given when the distribution of the input features (comparing training datasets VS prediction datasets) change enough (with a defined threshold) to raise an alert
  • target drift: distribution of the label values change when comparing training VS prediction distributions
  • concept drit: when the relation between the input features and target values change; it can arise when the label is redefined (for instance, the business rules for deciding clients who are active or inactive with some products; if the labeling rules changes, the same input feature values could be assigned to different target values before VS after redefining).

The way to monitor these drifts can be carried out via hypothesis testing as you say (e.g. Kolmogor-Smirnov test, Population Stability Index, etc), where you define the degree of warning thresholds.

The point is: what is the goal of having this drift monitoring? In general, the advice is to retraining your models when this type of drifts occur: it might or might not improve your model performance, but you make sure to have a model updated with fresh data. Other goal is also to have knowledge about your update data statistics of course.

Nevertheless, in this scenario of a subset of clients ages, although the model was trained with more "complete" datasets, you are making inference on a subset of the whole population used for training the model, so your model could still be valid enough (unless this new sceneario becomes the ususal one, so a more custom model could be trained with this new kind of more specific data).

$\endgroup$
2
  • $\begingroup$ In your last comment, are you saying that, suppose the trend of only targeting under 25s continues indefinitely (so all new data from now on are people under 25), it might be a good idea to retrain a new model on only customers that are under 25? Also, are you saying that if we have a very complete training data set, then it shouldn't matter if the new batch consists of people under 25 because they are represented by the training data? I think I understand my problem a lot better now so thank you! $\endgroup$ Commented Sep 14, 2022 at 13:13
  • 1
    $\begingroup$ Indeed to both questions. I mean, if the model generalized well enough and 25 years old people were included in the first training, it could still perform well predicting only on that segment. Nevertheless, I would try retraining in your second scenario, because it could be a more custom model for that segment of the population $\endgroup$
    – German C M
    Commented Sep 14, 2022 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.