I have a model running on a productive system. The model predicts if some lead will become a sale. How would you develop a check, which checks the success and the accuracy of the model? There is a time component because it took months where the lead could become a sale.

  • $\begingroup$ Why does it take months for a lead to become a sell? If the process between first-contact to sale is a complex process(which takes months to complete) I would predict first-contact success instead of predicting the sell. $\endgroup$ – yoav_aaa May 7 '20 at 9:03
  • $\begingroup$ Well, the important prediction is the first contact (lead) to sale. And after the lead is made, it takes time to make a sale. $\endgroup$ – Mutatos May 7 '20 at 9:19
  • $\begingroup$ And do you have data to support this saparation? I.E can you tell from your data the funnel of 'potential lead' >> first-contant event >> sell event? $\endgroup$ – yoav_aaa May 7 '20 at 9:21
  • $\begingroup$ I have the data, yes. $\endgroup$ – Mutatos May 7 '20 at 9:36

Short-term targets and more out-of-time samples.

Work with the business and your data to develop relevant short-term targets. For example, if a sale takes 6 months after first-contact and the sale is the modeled target, set first-contact as short-term target for measurement. Set other steps in the funnel as a short-term target. Then run your analysis with these. Similar to predicting a charge-off of a loan. If a model is used to give a loan and the loan is 2 years, we do not wait 2 years from model deployment to get the first day of measurements. That could be a lot of bad loans. Perhaps half of the people who typically charge off do so within 12 months. And then 75% charge off within 20 months. You need to examine what is relevant and how predictive those short-term targets are of the modeled target. Use your training data and SMEs.

Also, if your process takes time, then you did not use the most up-to-date data in modeling. For example, if a loan has a 24 month modeled target, the model could not have used the latest 24 months, since that does not have a mature target (assuming this is not a survivor model - then this paragraph does not hold). But now the model can be evaluated on that data as each month progresses. The model did not score that data for action, but the model can be evaluated. Perhaps some loans would not have been given with the new model, but this is better than waiting 2 years to determine that the model is bad.

  • $\begingroup$ Bu that means I don't need (I can't) to predict lead to sale in this case. I need to predict lead to first step, then predict first step to second step ..., last step to sale. Right? $\endgroup$ – Mutatos May 7 '20 at 10:34
  • $\begingroup$ You need to figure that out. If a sale takes 6 months after the model predicting who to target but you want a reading of the strength of the model earlier than 6 months, then you need to understand the steps in the business process to get to a sale and measure the model using that as a target for the analysis. Look at the funnel - what % of people who are targeted get to the 2nd step, the 3rd step, etc. Eventually you will have a fully mature target that gives you the sale. If this is an always on model, then you always have some mature and some immature to do analysis. $\endgroup$ – Craig May 8 '20 at 12:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.