How to select features when performing classification with a dataframe of multiple columns?

Question

I have a dataframe of 50000 observations and I want to perform a classification task. But I'm struggling with features selection. I have 89 columns, which after getting rid of some redundant features, I get down to 58 columns. Some of them are categorical, some numerical.

Any suggestions would be much appreciated.

Answer 1

You could use logistic regression with regulation (Lasso, Ridge, Elastic Net) to get an idea of what features are relevant. Regulation means, that features which are „not so important“ are shrunken. In R glmnet is a good starting point which is also available for Python.

R: https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html

Python: https://web.stanford.edu/~hastie/glmnet_python/

See Section 6.2 in Introduction to statistical learning (ISL) for some background and examples. You may find a PDF of the book online. http://www-bcf.usc.edu/~gareth/ISL/

Ridge is based on the L2 norm, so that model coefficients are shrunken, but cannot be zero. Lasso in contrast uses the L1 norm, so that coefficients can be shrunken to zero.

---

Another approach to model selection would be forward / backward stepwise selection (see: ISL, Ch. 6.1).

For forward stepwise selection, the process would be:

Let M0 be the NULL model, which containes no predictors. Now for k=0, ..., p-1:

1. Consider all p-k models that augment the predictors in M with one additional predictor.
2. Choose the best among these p-k models (e.g. based on smalles RSS / best R2).
3. Finally, select a single best model from the M models using cross-validated prediction error, AIC, BIC, or adjusted R2.

Backward stepwise selection works in the same way, but you start with the full model and successively leave out predictors/features.

For Python, sklearn comes with a greedy stepwise (backward) selection module: https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFE.html

---

Also note, that many methods/packages come with a module that allows to assess feature importance, e.g. in catboost (https://catboost.ai/docs/concepts/fstr.html) or lightgbm (https://lightgbm.readthedocs.io/en/latest/Python-API.html). However, a disadvantage here is, that you need to implement a proper model (with some hyperparameter tuning) first, which is time consuming.

Answer 2

According to me, feature selection is purely a manual process, though some algorithms like Randomforest, decisiontree provides you feature importance of the trained model.

Otherwise we have to identify the co-relation between features and the target values.You can draw the graph of features and the target value, in order to analyse the co-relation. You can also analyse the co-variance between features and the target.

JFI: PCA also internally sorts the values in decreasing order of co-variance.