I have been tasked to report on an ensemble model that was created in h2o which includes several model subtypes such as Random Forest, GBM, linear models etc. The end goal is to predict churn rates for products in a large telco company, but the approach we use could apply to any similar problem.
The models produced in this way contain a few potentially useful performance measures such as variable importance, precision, recall and some others. Each model has roughly 150 input variables.
The model scores have been used to group the customers by decile and measure the churn rate of each group.
The present situation is that the scores appear to be too good which suggests we may have a data leakage problem. For instance, for one of the models the 1st decile captures 84% of the churn, with 99% of the churn captured by the 4th decile.
My task is to understand and report on potential issues with the model performance so we can improve the models and recommend action to the business. What I would like to know is:
- What are some basic analyses that I can perform to address the data leakage issue.
- How can I leverage the model metadata to better understand model performance?
- What other important questions should I know to ask in order to fully address this task?