# How to perform Logistic Regression with a large number of features?

I have a dataset with 330 samples and 27 features for each sample, with a binary class problem for Logistic Regression.

According to the "rule if ten" I need at least 10 events for each feature to be included. Though, I have an imbalanced dataset, with 20% o positive class and 80% of negative class.

That gives me only 70 events, allowing approximately only 7/8 features to be included in the Logistic model.

I'd like to evaluate all the features as predictors, I don't want to hand pick any features.

So what would you suggest? Should I make all possible 7 features combinations? Should I evaluate each feature alone with an association model and then pick only the best ones for a final model?

I'm also curious about the handling of categorical and continuous features, can I mix them? If I have a categorical [0-1] and a continuous [0-100], should I normalize?

I'm currently working with Python.

Thanks a lot for your help!

• "I'm also curious about the handling of categorical and continuous features" I believe that would make a separate question. In fact, it's already been asked here. Jul 28, 2017 at 10:21
• there is a difference between not having enough samples and having irrelevant features. I wouldn't focus too much on picking exactly 7 features because of some simplistic rule...
– oW_
Jul 28, 2017 at 17:35
• Do what you'd do anyway: use cross-validation to optimize the regularization. I suggest elastic net (L1 + L2).
– Emre
Jul 28, 2017 at 21:17

In order to reduce your model down to 7 variables there are a few approaches you could take:

1. PCA (unsupervised): this creates "new" linear combinations of your data where each proceding component explains as much variance in the data as possible. So the first 7 components (out of 27) should be able to explain a good percentage of the variation in your data. You can then plug these seven components into your logistic regression equation. The disadvantage here is that because the components are combinations of your original variables you lose some interpretability with your regression model. It should however produce very good accuracy. This same technique applied to other dimension reduction methods such as
2. Another common method in regression is forward stepwise where you start with one variable and add on another each step, which is either kept or dropped based on some criteria (usually a BIC or AIC score). Backwards stepwise regression is the same thing but you start with all variables and remove one each time again based on some criteria. Based on a brief search it doesn't seem that python has a stepwise regression but they do a similar feature elimination algorithm described in this Data Science post.
3. Lasso Regression uses an $L_{1}$ penalization norm that shrinks the coefficients of features effectively eliminating some of them.You can include this $L_1$ norm into your logistic regression model. It seems sklearn's LogisticRegression allows you do assign the penalization you want in order to achieve this. Note: Lasso will not explicitly set variable coefficients to zero, but will shrink them allowing you to select the 7 largest coefficients.