Should highly correlated features be omitted before applying Lasso?

Question

I would greatly appreciate if you could let me know whether I should omit highly correlated features before using Lasso logistic regression (L1) to do feature selection.

In fact, I want to use logistic regression with L1 to do prediction as well as feature selection. However, some of my features are highly correlated e.g., -1 or 0.9. Should I omit them before applying Lasso or let the Lasso decide it?

Really, I read in Mr. Raschka’s book (Python Machine Learning) that

regularization is a very useful method to handle collinearity (high correlation among features).

However, this kernel (by referring to Wikipedia) states that keeping correlated features in the model would adversely affect the feature selection but it doesn't impair the predictions.

Answer 1

Use scikit-learn package. In your case you need find sklearn.linear_model.LogisticRegression

and User guide

It's clear enough for understanding. You needn't special actions to win collinearity. But instead of linear method you can use non parametric algorithm like random forest sklearn.ensemble.RandomForestClassifier.

Сompare the results of the logistic regression & random forest on the test data

Answer 2

there are different kinds of Regularization, you don't know what exactly the author meant... for huge multicollinearity problem - in statistics it is recommended to use Principal Component Analysis as unsupervised dimensionality reduction method. You need to do it prior classification with your model or use in sklearn Pipeline (doing pca first, then logistic regression)

and besides: as so as

"Regularization attempts to reduce the variance of the estimator by simplifying it, something that will increase the bias, in such a way that the expected error decreases."

-- that is: removing multicollinearity really aims to decrease the bias, but regularization leave the bias anyway - thus it is hard to say that regularization helps to cope with multicollinearity, just to prevent overfitting

objectives