Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I've created a model that has recently started suffering from drift.
I believe the drift is due to changes in the dataset but I don't know how to show that quantitatively.
What techniques are typically used to analyze and explain model (data) drift?
Extra:
The data is Tabular.
It depends about what type of data are we talking: tabular, image, text...
This is part of my PhD, so I am completely biased, I will suggest Explanation Shift. (I would love some feedback). It works well on tabular data.
In the related work section one can find other approaches.
The main idea under "Explanation Shift" is seeing how does distribution shift impact the model behaviour. By this we compare how the explanations (Shapley values) look on test set and on the supposed Out-Of-Distribution data.
The issue is that in the absence of the label of OOD data, (y_ood) one can not estimate the performance of the model. There is the need to either provide some samples of y_ood, or to characterize the type of shift. Since you can't calculate peformance metrics the second best is to understand how the model has changed.
There is a well known library Alibi https://github.com/SeldonIO/alibi-detect That has other methods :)
One way to start is fundamental exploratory data analysis.
Compare univariate, bivariate, and multivariate distributions between training data and new data. Those comparisons can be done visually, qualitatively, and quantitatively.
The exact methods would depend on the data type of the features of the tabular dataset. One specific example is the K-L divergence between two continuous distributions.
If you want to understand how different drift detection methods behave, here is a blog (with code) about the experiment when different drift detection methods were applied to artificially shifted datasets: https://www.evidentlyai.com/blog/data-drift-detection-large-datasets
Define comparison windows that match your use case and expected feature behavior: for example, you can compare your last-week data to the previous week, or all production data to validation data, or to some golden set, etc.
Evaluate prediction drift. It makes sense to separately look at the distribution shift in the model predictions (output drift) as it is often a great indicator that something has changed, e.g., it is predicting certain categories more often, etc.
Evaluate feature drift. You can perform per-feature drift detection and then measure the % of the drifted features and look at the drifted ones to visually explore/interpret what changed. Or only test top model features for drift.
You can use open-source libraries like Evidently https://github.com/evidentlyai/evidently that implement many drift detection methods and can quickly visualize distributions.
