You are unlikely to get a useful answer without a lot more details as there are lots of things that could cause this. How How many features and how many observations do you have?
It is possible that you have massively overfit your training set: Did you do a lot of hyper parameter tuning on your model? when you fit the light GBM, did you cross validate your results? Perhaps see what a 5 or 10 fold cross validation shows.
- Did you do a lot of hyper parameter tuning on your model?
- when you fit the light GBM, did you cross validate your results? Perhaps see what a 5 or 10 fold cross validation shows.
If you're predicting data about a future time point, test whether any of your features have massively changed between training and test sets. If i build a model on customer data from 2020, where my business only targetted small businesses, and in 2021 I went after much bigger customers, my model might not be very good at predicting things because the new data, which the model is tested on, has radically shifted.
- test whether any of your features have massively changed between training and test sets.
- If I build a model on customer data from 2020, where my business only targeted small businesses, and in 2021 I went after much bigger customers, my model might not be very good at predicting things because the new data, which the model is tested on, has radically shifted.
A few more things:
- ROC AUC is unlikely to be useful if your data is massively unbalanced (i.e. if <10% of customers churn). precision recall AUC is better, where your minority class is set to 1 in the model.
- Does your model give you a percentage of churning, and then you convert that percentage into "churn/no-churn"? If so, what threshold are you using? If you just unknowingly defaulted to "50%+ = churn
50%+ = churn, <50% no churn then consider whether that makes sense.
These are just a few ideas, there really is not substitute for exploring your data further though. Good luck!
