I am currently working on a research project where the central question is which features drive the prediction of different models. The main issue is, that there is high (multi-)collinearity among those features.

Imagine a setting with about 200 different features that are all potential candidates for helping predict the same dependent variable. In the past, the relevance of small sub-sets of like 5 of these features has simply been analyzed by throwing them into a linear regression model and analyzing R2 and significance of coefficients. In contrast, I want to evaluate feature importance for the complete set of 200 while also getting rid of the forced linear relationship in these regression models, i.e. I do not want to dictate any specific functional relationship ex ante. Thus, I train artificial neural networks and want to compare both their overall out-of-sample prediction performance and individual feature importance with the results from a multiple linear regression.

I am aware of model-independent measures for feature importance that allow comparing importance across different types of models (e.g. permutation importance). However, the issue comes from these measures being quite unreliable when there is high multicollinearity among features. For example, with two highly correlated features, the neural network will likely rely mainly on only one of the two predictors, even though both carry relevant predictive power for the given task. The same issues would of course also be present in the linear model. When you are just interested in your overall model performance, this would not really be an issue. However, my specific research question specifically deals with understanding which features carry relevant information for predicting the dependent variable (when not dictating any functional form ex-ante).

The only possible solution I have come up with so far, is specifying a feature's importance by taking maximum of its importance or any of the products of its correlation with another feature times that features importance (e.g. take the larger of the following two: feature importance of feature i vs. feature importance of feature j * correlation between i and j). But while I am not even sure if this is a "scientific" appropriate approach, it does also only adress the problem of pairwise correlation.

What would your go-to-approach be for creating a reliable measure for feature importance in the case of high (multi-)collinearity be? Can you maybe point me in the right direction of previous research on this topic that I might have overlooked?

Thanks for your help!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.