I have 200 variables with 200000 records. How to find best features from this variables? I have tried correlation technique via Heatmap but all the variables have near to same correlation score < 0.4 so it didn't worked.

Please suggest me which features selection technique will work?

  • $\begingroup$ You could use the scikit-learn library. It has a lot of feature selection algorithms that provide exactly that. $\endgroup$ – Giannis Krilis Feb 7 at 9:39

In model building there is a sort of iterative workflow that you can use:

  1. Select an appropriate model you want to build e.g. for classification maybe a XGB classifier or a logistic regression, etc.

This is important because the model by itself will determine a lot about how to wrangle your data. XGB only works with numerical features so you will have to convert factors/strings to a numerical encoding e.g. via One-Hot-Encoding.

  1. Build a full model using all features you can!

Some features will naturally fall out in the first step because the amount of feature extraction you have to do, to use them is too much to start. All other features, simply throw them into your model!

  1. Validate your model using classical validation methods (e.g. cross-validation, split-sample, etc.) and see how it performs!

If the performance is already great, perfect you are done! Otherwise you have a baseline against which to optimize the next steps.

  1. Play around with feature importance and removing features

Extract the feature importance from the full model and see if removing features with low importance improves your performance.

  1. Add features

At some point you will hit a wall in improving the model by simply removing irrelevant features (this might even be after removing 0 or 1 features). Now it is time to add by engineering additional features. Maybe now it is time to brush up your NLP skills and get some features out of the free-text variable you removed before.

  1. Rinse-and-repeat for other models to crown a winner in a model beauty contest
| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.