12
$\begingroup$

I am looking for tools that allow me to monitor machine learning models once they are gone to production. I would like to monitor:

  1. Long term changes: changes of distribution in the features with respect to training time, that would suggest retraining the model.
  2. Short term changes: bugs in the features (radical changes of distribution).
  3. Changes in the performance of the model with respect to a given metric.

I have been looking over the Internet, but I don't see any in-depth analysis of any of the cases. Can you provide me with techniques, books, references or Software?

| improve this question | | | | |
$\endgroup$
8
$\begingroup$

The changes in distribution with respect to training time are sometimes referred to as concept drift.

It seems to me that the amount of information available online about concept drift is not very large. You may start with its wikipedia page or some blog posts, like this and this.

In terms of research, you may want to take a look at the scientific production of João Gama, or at chapter 3 of his book.

Regarding software packages, a quick search reveals a couple of python libraries on github, like tornado and concept-drift.

| improve this answer | | | | |
$\endgroup$
  • $\begingroup$ +1, "concept drift detection" is probably a phrase to be searching for. As far as tools, I get the feeling most companies will have built in-house solutions so far; but now the DS-as-a-service companies are starting in, see AWS SageMaker Model Monitor. $\endgroup$ – Ben Reiniger Dec 23 '19 at 14:57
  • $\begingroup$ I finally got around to looking at the 2014 survey by Gama; it's really great. $\endgroup$ – Ben Reiniger Jan 3 at 17:03
1
$\begingroup$

What you're describing is known as concept drift and there are quite a few software startups bringing a solution to market (us included - happy to show you what we have).

  1. A very simplistic way of detecting drift is monitoring the differences between distributions of the predicted dataset and the training dataset using a Kolmogorov-Smirnov test or Wasserstein distance.

  2. For radical changes in distribution, what you might do is create a model to understand the datasets unique patterns and have an outlier detector to determine true radical changes to the distribution as opposed to also identifying false positives.

  3. This is an interesting use case - are you able to share an example?

| improve this answer | | | | |
$\endgroup$
0
$\begingroup$

If I understand your query correctly,you are looking for MLFLOW where you can track your experimentation and vizualize them using APIs MLFLOW

| improve this answer | | | | |
$\endgroup$
0
$\begingroup$

Something like this:

All credits goes to Abishek in any case as long as you save your desired differences, metrics, changes locally in your source code, you can hook it up with slack and receive messages. So in your cases for all 3 things its totally doable, you just have to get your hands dirty a bit (looks like a great hobby project!)

enter image description here

| improve this answer | | | | |
$\endgroup$
0
$\begingroup$

As mentioned, there are a few solutions that emerged in the last 18 months. I am the founder of superwise.ai - happy to elaborate more on that, and you're invited to check our "model healthcheck" (https://superwise.ai/healthcheck/) one-off service to get an idea of the insights you can get from our software.

All the best Ofer Razon

| improve this answer | | | | |
$\endgroup$
0
$\begingroup$

You can have a look at Anodot's MLWatcher. Few of the highlights of this tool are as follows.

  • MLWatcher collects and monitors metrics from machine learning models in production.
  • This open source Python agent is free to use, simply connect to your BI service to visualize the results.
  • To detect anomalies in these metrics, either set rule-based alerting, or sync to an ML anomaly detection solution, such as Anodot, to execute at scale.
  • Distribution of input features.

You can have a look at their complete features here.

| improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.