In general, there is no clear and easy way for deciding which features to include in a model.
That being said, there are different strategies you can use to process features in an efficient way:
- Domain knowledge
One of the most important aspects when determining important features is the knowledge of the specific domain related to your dataset. This might mean reading past research papers that have explored similar topics or asking key stakeholders to determine what they believe the most important factors are for predicting the target variable.
So, these methods depend on your knowledge from the domain.
- Wrapper methods
These methods determine the optimal subset of features using different combinations of features to train models and then calculating performance. Every subset is used to train models and then evaluated on a test set. Sp, these methods could end up being very computationally intensive, though they are highly effective in determining the optimal subset. It is not suggested to use these methods with large feature sets because they are computationally expensive.
For this case you can use Recursive Feature elimination (RFE)
- Filter methods
In these methods, feature selection methods carried out as a pre-processing step before even running a model.
They work by observing characteristics of how features are related to one another. Different metrics can be used to determine which features will get eliminated and which will remain. They also return a feature ranking that tells you how independent variables are ordered in relation to one another. They will remove the variables that are considered redundant.
One example of this method is "VarianceThreshold" from sklearn
- Embedded methods
These methods are included within the actual formulation of the machine learning algorithm. The most common type of embedded method is regularization such as lasso.