Assume I have a model which predicts the outcome of the number of icecreams sold in a store.

The model is trained on data for the last 5 years while keeping the last year as a validation set and has produced very good results.

We now put the model into production such that the CFO can create an estimate for the upcoming year's budget. The CFO now look at the prediction for May, say 2000 ice creams, and thinks "Ooh... I was hoping for some more sale in May. I'll go 4000" thus he orders some more advertising, introduces new flavors, etc. and reaches the 4000 sold ice cream at the end of May as he was hoping for.

On the first of June, we talk to the CFO to evaluate the model after the first 6 months, and we see that our prediction in May is off by 100%!

This spike can be explained with the increased advertising etc., and all the other days the model has done really well, but if the CFO starts tweaking the advertising, flavors, etc. each day to hit the budget, how will we ever be able to test, if our model is indeed good in production/real-world? And how will we be able to re-train the model, since the first 5 years sale is without any "human influence" whereas, after a year, the sale has been influenced by advertising, etc., thus the spike in May is not "natural" but is due to some exogenous variable we are not able to incorporate (e.g we don't know the CFO's budget)?


2 Answers 2


I am afraid that such situations are fundamentally inherent in predicting/forecasting contexts; quoting from the very recent paper by Taleb et al., On single point forecasts for fat-tailed variables (open access, para 3.7):

3.7. Forecasts can result in adjustments that make forecasts less accurate

It is obvious that if forecasts lead to adjustments, and responses that affect the studied phenomenon, then one can no longer judge these forecasts on their subsequent accuracy.

So, other than communicating this clearly beforehand and reaching an agreement on how the predictions will be assessed in the presence of such adjustments, there is not much else you can do from a modelling or methodology perspective. The advice suggested further in the same paragraph quoted above:

In that sense a forecast can be a warning of the style “if you do not act, these are the costs”.

can form the basis of such a communication and agreement.


The simpler, more practical, and more business-oriented way to go would be to include advertising in your model. That would allow you to :

  • Change your prediction accordingly, then your measured performance. The key is to be transparent about it, pedagogical even. Basically, in your example, you had 6 months to review your prediction and inform the CFO so that he doesn't think your model is off. If the problem is the absence of communication on his part, you can formulate it that way: more communication from his side would allow for better predictions. If he is not willing to do so that's on him.

  • Help evaluate the advertising needed. With a simple model, you could open some interesting discussions with your CFO. Basically, if you predict 2000 and he wants to sell 4000 you can try to get what change in the advertising feature would lead to a 2000 sale increase. It would allow you to discuss when advertising is worth it and it might open the discussion about the first point (prediction update).

  • $\begingroup$ The problem is you do not know in advance if there will be increasing advertisement etc, it depends on your prediction i.e if the prediction is too low then there will be more advertisement etc, thus it is difficult to include it as a feature $\endgroup$
    – CutePoison
    Commented Oct 20, 2020 at 5:00
  • 1
    $\begingroup$ Imo the underlying dynamic of spending more money doesn't really matter. What matter is the behavior of consumer under advertising. Build a model with past advertising as a feature. Give an initial prediction without additional spending. Modify the advertising feature as they update spending. $\endgroup$ Commented Oct 20, 2020 at 6:25
  • 1
    $\begingroup$ Or in other words : it's very difficult (and it's probably not your job) to modelise upper management decisions. What you can usually do is to provide them tools to take better decisions. $\endgroup$ Commented Oct 20, 2020 at 6:29
  • $\begingroup$ I agree on that! The issue is actually how to validate the model in production, since the space it is trained on is not the same as it is tested on. $\endgroup$
    – CutePoison
    Commented Oct 20, 2020 at 6:40
  • $\begingroup$ Well the main practical problem is that you have to keep track of the additional spending and update your prediction and evaluate the new prediction. For the theoretical problem you mention, I don't think there is a more practical solution than assuming your consumer behavior model is robust enough that it won't crumble when you updtae one of the parameters. I can agree that's not optimal (especially if you have correlation) but it's practical. At the end of the day you would just have to check if the spending update and prediction aren't unusual. $\endgroup$ Commented Oct 20, 2020 at 6:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.