I am looking for tools that allow me to monitor machine learning models once they are gone to production. I would like to monitor:

  1. Long term changes: changes of distribution in the features with respect to training time, that would suggest retraining the model.
  2. Short term changes: bugs in the features (radical changes of distribution).
  3. Changes in the performance of the model with respect to a given metric.

I have been looking over the Internet, but I don't see any in-depth analysis of any of the cases. Can you provide me with techniques, books, references or Software?


The changes in distribution with respect to training time are sometimes referred to as concept drift.

It seems to me that the amount of information available online about concept drift is not very large. You may start with its wikipedia page or some blog posts, like this and this.

In terms of research, you may want to take a look at the scientific production of João Gama, or at chapter 3 of his book.

Regarding software packages, a quick search reveals a couple of python libraries on github, like tornado and concept-drift.

  • $\begingroup$ +1, "concept drift detection" is probably a phrase to be searching for. As far as tools, I get the feeling most companies will have built in-house solutions so far; but now the DS-as-a-service companies are starting in, see AWS SageMaker Model Monitor. $\endgroup$ – Ben Reiniger Dec 23 '19 at 14:57
  • $\begingroup$ I finally got around to looking at the 2014 survey by Gama; it's really great. $\endgroup$ – Ben Reiniger Jan 3 at 17:03

Something like this:

All credits goes to Abishek in any case as long as you save your desired differences, metrics, changes locally in your source code, you can hook it up with slack and receive messages. So in your cases for all 3 things its totally doable, you just have to get your hands dirty a bit (looks like a great hobby project!)

enter image description here


If I understand your query correctly,you are looking for MLFLOW where you can track your experimentation and vizualize them using APIs MLFLOW


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.