Doing simple supervised regression where the label is a floating point number (guaranteed positive) and the features are a mix of continuous floating point values and some categorical features.
What would be a good metric of global bivariate dependence that would assign a value to each feature, based on the training dataset. By that I am trying to say
- The importance will be calculated for each feature, not each feature and each sample
- Best if all of them are positive and add up to one, which can be easily achieved by doing a SoftMax as the final step after getting the values otherwise.
- The importance should not be dictated by any specific model, such as CatBoost or XGBoost etc. Just a measure based on the data alone.
If all the features were continuous floats, I would just take the Pearson correlation, but some are categorical as I pointed out. Even for categorical, likely measures like mutual information or entropy can help, but then it is difficult to compare the numbers among features.