Decision trees are by nature immune to multi-collinearity. So by that principal so is Random Forest. For example, if you have 2 features which are 99% correlated, when deciding upon a split the tree will choose only one of them. Other models such as Logistic regression would use both the features. - follow this link to know more.
For other models, solutions can be many, but here are the popular choices -
Regularization
This is an automatic way of handling high correlations in your Xs. It decreases the value of the co-efficients by penalizing them against the loss function and introduces bias in the system. Popular implementations are L1, L2 and Elastic Net(in case of linear and logistic models), Pruning(in case of Decision Trees), Dropout and max pooling in case of deep learning.
Almost all sklearn implementations support a regularization parameter in 1 way or the other. Depends on the algorithm you are using.
Using VIF to detect multi-collinearity
Variance Inflation Factor is a metric that you can use with all your Xs and then start dropping variables with the highest VIF one by one (usually the variable with the highest VIF is dropped and then the exercise is repeated till there is no multi-collinearity left or domain experts come in and choose what important variables to keep/drop)
Principal Component Analysis
Although the model might lose its interpretability (you won't be able to tell exactly what feature is how important), PCA has been another effective way of removing multi-collinearity. Basically it projects the data in a way that the output columns are orthogonal (independent)
Some Examples
Ridge Regression
from sklearn.linear_model import Ridge
import numpy as np
n_samples, n_features = 10, 5
np.random.seed(0)
y = np.random.randn(n_samples)
X = np.random.randn(n_samples, n_features)
clf = Ridge(alpha=1.0)
clf.fit(X, y)
Lasso Regression
from sklearn import linear_model
clf = linear_model.Lasso(alpha=0.1)
clf.fit([[0,0], [1, 1], [2, 2]], [0, 1, 2])
Elastic Net
from sklearn.linear_model import ElasticNet
from sklearn.datasets import make_regression
X, y = make_regression(n_features=2, random_state=0)
regr = ElasticNet(random_state=0)
regr.fit(X, y)
Random Forest
- Almost all parameters in scikit-learn can account for regularization here
Statsmodels has a rather good implementation of VIF
PCA
import numpy as np
from sklearn.decomposition import PCA
from sklearn.svm import SVC
X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
pca = PCA(n_components=2) # adjust yourself
pca.fit(X)
X_t_train = pca.transform(X)
clf = SVC()
clf.fit(X_t_train, y)
