Recently, I do 2 different ML projects.
One is lending club loan prediction, another is a pravite dataset in online experiment field to predict whether a customer will take the treatment.
Both 2 tasks are binary classification with 100+Million observations and a hundred covariates. However, my model of lending club have a very high PR-ROC(0.86), which shows the good performance of model. My model of online experiment suffers, only have a 0.03 PR-AUC. The model is somekind useless.
I tried to explain to my leader that the dataset is so uninformative and that's the real reason why my project failed. I use low metric scores(PR-AUC = 0.03) and high loss function values as proofs to show the model I build is useless.
Later, I came across one question, how do we measure the information contains in the covariates regardless of the model we built?
If we perform regression tasks using linear model, we can use
BIC to select a model, choosing the best model who mined the covariates best. If we perform binary classification ML task like I did before, we can use
PR-ROC metrics etc. I think these metrics helps us to compare model performance instead of the potential data quality.
The solution I want is something like entropy. For instance, we can calculate the entropy between 2 probability distributions or between covariate and target label. Entropy shows us the relation intensity between covariates and label. Is there any better solution to measure the information contains in the covariates?
I am a graduate of statistics, thanks a lot if you guys can provide me any source to learn!