How to approach model reporting task

Question

I have been tasked to report on an ensemble model that was created in h2o which includes several model subtypes such as Random Forest, GBM, linear models etc. The end goal is to predict churn rates for products in a large telco company, but the approach we use could apply to any similar problem.

The models produced in this way contain a few potentially useful performance measures such as variable importance, precision, recall and some others. Each model has roughly 150 input variables.

The model scores have been used to group the customers by decile and measure the churn rate of each group.

The present situation is that the scores appear to be too good which suggests we may have a data leakage problem. For instance, for one of the models the 1st decile captures 84% of the churn, with 99% of the churn captured by the 4th decile.

My task is to understand and report on potential issues with the model performance so we can improve the models and recommend action to the business. What I would like to know is:

1. What are some basic analyses that I can perform to address the data leakage issue.
2. How can I leverage the model metadata to better understand model performance?
3. What other important questions should I know to ask in order to fully address this task?

Answer 1

Remove input data to test for leakage

This is very generalized question, so without knowing the types and provenance of the input data, this can be hard to answer.

But, in general, to check for leakage, you can use the model on some subsets of the input variables while removing other input variables. If you get data from multiple sources, then try removing all input variables from a single source, then re-run your models. You may be able to identify the source of the data leakage. Alternately, if computational power allows, you can brute force it by running the model with each of the 150 input variables removed, or all sets of two variables, etc.

Use customer-centered time data

Regarding model meta-data, again I would investigate data provenance. Are you predicting churn using the complete patterns of customers who stopped using the service? What I mean to say is, instead of looking backwards from a fixed real-time period, like today, to all customers who did or did not stop using the service, try looking from a fixed customer-time. Use only data from the first year that each customer used the service, and attempt to predict whether each customer will remain with the service for another year.

The warning signs of a customer dropping the service may be obvious in the months leading up to that customer dropping the service, but by then, the predictive power of your model may be too late to stop that customer from leaving. Instead, index the time component of each customer's history to zero when the first start using the service, and run your model on this data.