How to perform Logistic Regression with a large number of features?

Question

I have a dataset with 330 samples and 27 features for each sample, with a binary class problem for Logistic Regression.

According to the "rule if ten" I need at least 10 events for each feature to be included. Though, I have an imbalanced dataset, with 20% o positive class and 80% of negative class.

That gives me only 70 events, allowing approximately only 7/8 features to be included in the Logistic model.

I'd like to evaluate all the features as predictors, I don't want to hand pick any features.

So what would you suggest? Should I make all possible 7 features combinations? Should I evaluate each feature alone with an association model and then pick only the best ones for a final model?

I'm also curious about the handling of categorical and continuous features, can I mix them? If I have a categorical [0-1] and a continuous [0-100], should I normalize?

I'm currently working with Python.

Thanks a lot for your help!

Answer 1

In order to reduce your model down to 7 variables there are a few approaches you could take:

1. PCA (unsupervised): this creates "new" linear combinations of your data where each proceding component explains as much variance in the data as possible. So the first 7 components (out of 27) should be able to explain a good percentage of the variation in your data. You can then plug these seven components into your logistic regression equation. The disadvantage here is that because the components are combinations of your original variables you lose some interpretability with your regression model. It should however produce very good accuracy. This same technique applied to other dimension reduction methods such as
2. Another common method in regression is forward stepwise where you start with one variable and add on another each step, which is either kept or dropped based on some criteria (usually a BIC or AIC score). Backwards stepwise regression is the same thing but you start with all variables and remove one each time again based on some criteria. Based on a brief search it doesn't seem that python has a stepwise regression but they do a similar feature elimination algorithm described in this Data Science post.
3. Lasso Regression uses an $L_{1}$ penalization norm that shrinks the coefficients of features effectively eliminating some of them.You can include this $L_1$ norm into your logistic regression model. It seems sklearn's LogisticRegression allows you do assign the penalization you want in order to achieve this. Note: Lasso will not explicitly set variable coefficients to zero, but will shrink them allowing you to select the 7 largest coefficients.

As @E_net4 commented, your continuous question is addressed in another post.

Answer 2

You're taking the "Rule of 10" too seriously. It's a very rough rule of thumb. It's not intended to be used like you are using it.

It sounds like you are thinking: "I have only 70 positive instances, so by the Rule of 10, I'm only allowed to use 7 features; how do I choose which 7 features to use?"

That's not what the Rule of 10 means. It's not some rule that specifies how many features you are permitted to use. The Rule of 10 is descriptive, not prescriptive, and it's an approximate guideline: if the number of instances is much fewer than 10 times the number of features, you're at especially high risk of overfitting, and you might get poor results.

So what should you do? You should do what you'd do anyway: use regularization, and use cross-validation to select the regularization hyper-parameters. Also, it's important to have a hold-out test set that you don't touch until you've finalized everything about the classifier, to avoid overfitting and biased accuracy estimates.

And if you can get more data, that would really help.

Finally, since you have imbalanced classes, you might consider reading about class imbalance and methods for dealing with it.