How to do feature selection or feature engineering in datasets with a lot of features?

Question

To make a good ML model, we have to select features that increase model accuracy and, if needed, to "engineer" features (e.g. apply some function like logarithm or square to linear regression predictor if it doesn't linearly correlate with predicted variable). But how can I do that to a large data set with a lot of features? Shall I test every single variable for different conditions, or there are some simpler ways to do feature selection or feature engineering.

Answer 1

Starting by saying that it is a really wide question and that without knowledge of the specific problem you have in mind it is difficult to give you a general answer, one thing you can do is to apply PCA, so you can start to study which of the many features you have explains more variance.

Then having done such a rotation (i.e. you have re-written your features on the eigenvectors of the covariance operator), you can start reducing your features if you want, or focusing your attention only on the most important ones, also in order to find (higher-order) correlations.

Then a general "technique" I apply is the following: Do feature engineering as a second step, leaded by error analysis, that is train a baseline model and perform error analysis, only like this you will be able to find the right operations that actually will improve your model performances.