How to measure the information of covariates in a ML task?

Question

Background

Recently, I do 2 different ML projects.

One is lending club loan prediction, another is a pravite dataset in online experiment field to predict whether a customer will take the treatment.

Both 2 tasks are binary classification with 100+Million observations and a hundred covariates. However, my model of lending club have a very high PR-ROC(0.86), which shows the good performance of model. My model of online experiment suffers, only have a 0.03 PR-AUC. The model is somekind useless.

I tried to explain to my leader that the dataset is so uninformative and that's the real reason why my project failed. I use low metric scores(PR-AUC = 0.03) and high loss function values as proofs to show the model I build is useless.

Question

Later, I came across one question, how do we measure the information contains in the covariates regardless of the model we built?

If we perform regression tasks using linear model, we can use RMSE, AIC, BIC to select a model, choosing the best model who mined the covariates best. If we perform binary classification ML task like I did before, we can use F1, ROC and PR-ROC metrics etc. I think these metrics helps us to compare model performance instead of the potential data quality.

The solution I want is something like entropy. For instance, we can calculate the entropy between 2 probability distributions or between covariate and target label. Entropy shows us the relation intensity between covariates and label. Is there any better solution to measure the information contains in the covariates?

I am a graduate of statistics, thanks a lot if you guys can provide me any source to learn!

Answer 1

There exists a notion of entropy in information theory. It is in fact closely linked to covariance, and especially the covariance matrix determinant (I don't recall the actual mathematical link but I think this should be something that Google can find).

From then, you could use some metrics based on the dataset correlation matrix. You could use its determinant (low determinant means highly correlated data), but it's not always easy to analyse (2 perfectly correlated features will drop the determinant value to 0). The distribution of correlation matrix eigenvalues will surely give more detailed insight (multiple low eigenvalues will mean lots of correlated features).

Another powerful and less mathematical solution would be to compute PCA over the dataset. Give yourself a threshold of explained variance (for instance 95%), and count how many principal components you need to reach this explained variance. A low number of such would mean low "quality" data.