Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I've created a model that has recently started suffering from drift.
I believe the drift is due to changes in the dataset but I don't know how to show that quantitatively.
What techniques are typically used to analyze and explain model (data) drift?
Extra:
The data is Tabular.
It depends about what type of data are we talking: tabular, image, text...
This is part of my PhD, so I am completely biased, I will suggest Explanation Shift. (I would love some feedback). It works well on tabular data.
In the related work section one can find other approaches.
The main idea under "Explanation Shift" is seeing how does distribution shift impact the model behaviour. By this we compare how the explanations (Shapley values) look on test set and on the supposed Out-Of-Distribution data.
The issue is that in the absence of the label of OOD data, (y_ood) one can not estimate the performance of the model. There is the need to either provide some samples of y_ood, or to characterize the type of shift. Since you can't calculate peformance metrics the second best is to understand how the model has changed.
There is a well known library Alibi https://github.com/SeldonIO/alibi-detect That has other methods :)
One way to start is fundamental exploratory data analysis.
Compare univariate, bivariate, and multivariate distributions between training data and new data. Those comparisons can be done visually, qualitatively, and quantitatively.
The exact methods would depend on the data type of the features of the tabular dataset. One concrete example is the K-L divergence between two continuous distributions.
