Problem statement: Given a set of random variables Xs, give the order of importance of their ability to explain variance in target variable Y. Assumes all Xs are normalised to have 0 mean and std dev of 1.

Can treat this as regression problem(normal or ordinal)

Solution for 2 cases: Case 1: When the calculated covariance matrix is diagonal, I know that random variables Xs are independent wrt each other. Then I can fit a linear regression model and order Xs by their weights.

Case 2: Or If I know that Xs are independent(https://en.wikipedia.org/wiki/Independence_(probability_theory) even though their covariance in non zero, I can fit linear regression model and order Xs by weights.

How do I solve this problem when I know that random variable Xs are not independent and have non zero covariance?

Also how can I use a non linear model such as random forest to get variable importance? scikit provides feature_importances_ api for random forest.... Is there an gotcha with this api such as results are not meaningful when variables are dependent or covariant?

Found this question elsewhere: https://stats.stackexchange.com/questions/202221/for-linear-classifiers-do-larger-coefficients-imply-more-important-features/202846


1 Answer 1


Decision trees are by nature immune to multi-collinearity. So by that principal so is Random Forest. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models such as Logistic regression would use both the features. - follow this link to know more.

For other models, solutions can be many, but here are the popular choices -


This is an automatic way of handling high correlations in your Xs. It decreases the value of the co-efficients by penalizing them against the loss function and introduces bias in the system. Popular implementations are L1, L2 and Elastic Net(in case of linear and logistic models), Pruning(in case of Decision Trees), Dropout and max pooling in case of deep learning.

Almost all sklearn implementations support a regularization parameter in 1 way or the other. Depends on the algorithm you are using.

Using VIF to detect multi-collinearity

Variance Inflation Factor is a metric that you can use with all your Xs and then start dropping variables with the highest VIF one by one (usually the variable with the highest VIF is dropped and then the exercise is repeated till there is no multi-collinearity left or domain experts come in and choose what important variables to keep/drop)

Principal Component Analysis

Although the model might lose its interpretability (you won't be able to tell exactly what feature is how important), PCA has been another effective way of removing multi-collinearity. Basically it projects the data in a way that the output columns are orthogonal (independent)

Some Examples

Ridge Regression

 from sklearn.linear_model import Ridge
 import numpy as np
 n_samples, n_features = 10, 5
 y = np.random.randn(n_samples)
 X = np.random.randn(n_samples, n_features)
 clf = Ridge(alpha=1.0)
 clf.fit(X, y) 

Lasso Regression

 from sklearn import linear_model
 clf = linear_model.Lasso(alpha=0.1)
 clf.fit([[0,0], [1, 1], [2, 2]], [0, 1, 2])

Elastic Net

 from sklearn.linear_model import ElasticNet
 from sklearn.datasets import make_regression

 X, y = make_regression(n_features=2, random_state=0)
 regr = ElasticNet(random_state=0)
 regr.fit(X, y)

Random Forest - Almost all parameters in scikit-learn can account for regularization here

Statsmodels has a rather good implementation of VIF


import numpy as np
from sklearn.decomposition import PCA
from sklearn.svm import SVC

X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
pca = PCA(n_components=2) # adjust yourself

X_t_train = pca.transform(X)
clf = SVC()
clf.fit(X_t_train, y)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.