In model building there is a sort of iterative workflow that you can use:
- Select an appropriate model you want to build e.g. for classification maybe a XGB classifier or a logistic regression, etc.
This is important because the model by itself will determine a lot about how to wrangle your data. XGB only works with numerical features so you will have to convert factors/strings to a numerical encoding e.g. via One-Hot-Encoding.
- Build a full model using all features you can!
Some features will naturally fall out in the first step because the amount of feature extraction you have to do, to use them is too much to start. All other features, simply throw them into your model!
- Validate your model using classical validation methods (e.g. cross-validation, split-sample, etc.) and see how it performs!
If the performance is already great, perfect you are done! Otherwise you have a baseline against which to optimize the next steps.
- Play around with feature importance and removing features
Extract the feature importance from the full model and see if removing features with low importance improves your performance.
- Add features
At some point you will hit a wall in improving the model by simply removing irrelevant features (this might even be after removing 0 or 1 features). Now it is time to add by engineering additional features. Maybe now it is time to brush up your NLP skills and get some features out of the free-text variable you removed before.
- Rinse-and-repeat for other models to crown a winner in a model beauty contest