How do you evaluate ML model already deployed in production?

Question

so to be more clear lets consider the problem of loan default prediction. Let's say I have trained and tested off-line multiple classifiers and ensembled them. Then I gave this model to production.

But because people change, data and many other factors change as well. And performance of our model eventually will decrease. So then it needs to be replaced with the new, better model.

What are the common techniques, model stability tests, model performance tests, metrics after deployment? How to decide when to replace current model with newer one?

Answer 1

What you should consider more often in production scenario is revenue for your model , and A/B test is a must .

As in your case , you can exactly measure how much money can your model for loan default prediction bring to you , or how much loss can your model can save for you .

Besides , you can check if the distribution of your prediction is consistent with that of ground truth concerning accuracy and stability for your model .

Hopes this will help you , good luck -)

Answer 2

After the models are deployed in production, I'd monitor the following:

(1) The same metric you used to evaluate the performance of your model, in some cases it is accuracy, or it could be precision, recall, RMSE. I'd plot a daily time series charting the metric and see that it is still performing above a satisfactory threshold. There might be seasonality within the calender, the model performs well around certain months and not so well in other months. I'd compare the performance against the test/validation sets of the same months to account for seasonality.

(2) Apart from looking at the performance of the model, especially if one is working with shared computing resources, I'd also recommend keeping a close tab on the data aggregation runtime, model runtime, success rates of models runs over the past period.

Answer 3

This question is very common in automation, when machine learning used to perform specific tasks. Guaranteeing the quality is always a must. Evaluating the model while it is in production is not an easy task. the reason, why? In order to evaluate the model in production you need to have the ground truth. This ground truth is not available (if it is available no need to have the model). Getting the ground truth (by using human for example) is not a good solution: 1- it is very expensive, 2- again if you will generate the ground truth for data in production then no need to have a model. BUT how to deal with this problem in reality? I have worked recently on a prediction model that is used to predict the vehicles (Makes, Models), since every year we may have new models, makes, it is a good question to ask how often do I have to repeat the training process? Three different ways I used to answer this question: 1. I analyzed the changing on the training data I have year by year. based on this variation I can estimate the number of new makes and models appear every year and the number of makes and models disappear every year, and thus i can estimate the expected degradation in performance. 2. I did several experiments using the data from 1990-2014 to predict 2015. Using Data from 1991-2015 to predict 2016. This helps me a lot in understanding how much my model is invariant from year to year. 3. Instead of scanning all the data from production, you can randomly sample. The used distribution can be adaptive, such that the number of sampled records increases gradually by time. The reason why the distribution is adaptive, because we are expecting that the model performance will deviate from the the expected performance with time.