I have been tasked to report on an ensemble model that was created in h2o which includes several model subtypes such as Random Forest, GBM, linear models etc. The end goal is to predict churn rates for products in a large telco company, but the approach we use could apply to any similar problem.

The models produced in this way contain a few potentially useful performance measures such as variable importance, precision, recall and some others. Each model has roughly 150 input variables.

The model scores have been used to group the customers by decile and measure the churn rate of each group.

The present situation is that the scores appear to be too good which suggests we may have a data leakage problem. For instance, for one of the models the 1st decile captures 84% of the churn, with 99% of the churn captured by the 4th decile.

My task is to understand and report on potential issues with the model performance so we can improve the models and recommend action to the business. What I would like to know is:

  1. What are some basic analyses that I can perform to address the data leakage issue.
  2. How can I leverage the model metadata to better understand model performance?
  3. What other important questions should I know to ask in order to fully address this task?
  • $\begingroup$ Are you getting 99% prediction on cross validated data? An explanatory model run on all data can often be very accurate, but if you spit the dataset in some way you may find that in cross validation it is relatively poor at prediction. $\endgroup$
    – kingledion
    Feb 20, 2018 at 15:46
  • $\begingroup$ The churn deciles I describe above come from our 'test' data set. All of our development and evaluation has been done using training/test data sets. We could have further split into training/test/dev to avoid additional bias, but honestly there was not a lot of iteration done to optimize these models. They perform this well using mostly default settings in h2o. $\endgroup$
    – Sledge
    Feb 20, 2018 at 15:57

1 Answer 1


Remove input data to test for leakage

This is very generalized question, so without knowing the types and provenance of the input data, this can be hard to answer.

But, in general, to check for leakage, you can use the model on some subsets of the input variables while removing other input variables. If you get data from multiple sources, then try removing all input variables from a single source, then re-run your models. You may be able to identify the source of the data leakage. Alternately, if computational power allows, you can brute force it by running the model with each of the 150 input variables removed, or all sets of two variables, etc.

Use customer-centered time data

Regarding model meta-data, again I would investigate data provenance. Are you predicting churn using the complete patterns of customers who stopped using the service? What I mean to say is, instead of looking backwards from a fixed real-time period, like today, to all customers who did or did not stop using the service, try looking from a fixed customer-time. Use only data from the first year that each customer used the service, and attempt to predict whether each customer will remain with the service for another year.

The warning signs of a customer dropping the service may be obvious in the months leading up to that customer dropping the service, but by then, the predictive power of your model may be too late to stop that customer from leaving. Instead, index the time component of each customer's history to zero when the first start using the service, and run your model on this data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.