This seems to be a common problem when bringing machine learning models to production.

Let's say we have an optimized machine learning model which gives decent performance metric in the unseen testing dataset. We are quite satisfied with that, and decided to bring the model online. Then we use A/B test to compare our website performance (i.e., revenue, customer engagement etc) with and without the new model. Somehow, our new model is not a clear winner or even a clear loser in the A/B test. How do we deal with such situation?

Here the model I mentioned is a machine learning model, for example ranking algorithm or a recommendation algorithm, but can be any algorithm in reality. Thanks for any help!

  • $\begingroup$ In this scenario the optimized ML model replaces an older ML model right? That is, the A/B test compares old model A vs. new model B? If yes, does that mean that old model A performed significantly worse than B on the offline data? $\endgroup$
    – Erwan
    Oct 3, 2020 at 0:42
  • $\begingroup$ @Erwan Yes, correct. So A performs better than B in the offline evaluation stage, while worse than B in the online evaluation stage. A is trained using more recent data than B. So what should we do here? $\endgroup$
    – CathyQian
    Oct 15, 2020 at 14:35
  • 3
    $\begingroup$ So apparently there's a discrepancy between the data collected online and the offline testing data (and probably also the recent training data used for model A). It seems to me that this difference needs to be investigated: apparently the offline data is not representative enough of the online data. A few hypotheses come to mind: maybe there has been a drastic change in customers habits between the moment the offline data was collected and now? (covid19 changed a lot of things); or maybe it's a matter of amount of data, for instance if the offline data was collected ... $\endgroup$
    – Erwan
    Oct 15, 2020 at 18:50
  • 2
    $\begingroup$ .... during a too short period of time and it's not representative of more general trends. Or maybe it's the online data collected for A/B testing which is too small and not representative enough of the general distribution. $\endgroup$
    – Erwan
    Oct 15, 2020 at 18:50

1 Answer 1


One way to deal with the situation is to investigate the differences between the training and A/B testing. Here a couple of common differences:

  • The modeling training process optimizes a machine learning loss function. A/B test optimizes a business value. The loss function and business value could diverge.

  • Data distributions are different. The machine learning model is trained on older data. The A/B test is on newer data. The older and newer data come from different distributions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.