Correlation based Feature Selection vs Feature Engineering

Question

I'm a bit confused about the superiority of Feature Selection over Feature Engineering or vice versa.

Let's say I just want to get the best possible performance on a couple of models like a neural network, something tree-based and a Naive Bayes Classifier.

Before starting any training I looked at my features and engineered some additional (hopefully) even more expressive features. I did this from a domain expert point of view. For instance I added a new ratio feature C = A / B because I think this will be a very expressive information for the model.

Furthermore I added several features measuring basically the same thing but in different ways. Lets say a feature D measuring the the length of any text including empty lines and an additional feature E measures the length of any text excluding empty lines.

So this leads to very many features in my data set with also very high correlation / multi-collinearity. (of course D and E are very high correlated and A, B and C are also high correlated / multi-collinear.

So any kind of correlated based feature selection (between the features) would remove a lot of the engineered features, but could the removal provide the model any kind of better discriminative power with just less information? What helps the model more, keeping all features or removing correlated ones?

Answer 1

What helps the model more, keeping all features or removing correlated ones?

• There is some theory about it but in the end Machine Learning is try and error. You should give it a try with all features and then doing a feature selection to see if you are able to improve your model. What works for some models doesn´t necessarily have to work for the rest of the models.

If you want to select which features help your model you could do otherwise, instead of having all and removing, starting with one and adding features and only keeping them in case that it boosts the performance of the model. There are cases when you add a feature and the performance of the model drops.

There are a lot of ways in which we can think of feature selection, but most feature selection methods can be divided into three major buckets. From this source

• Filter based: We specify some metric and based on that filter features. An example of such a metric could be correlation/chi-square.
• Wrapper-based: Wrapper methods consider the selection of a set of features as a search problem. Example: Recursive Feature Elimination
• Embedded: Embedded methods use algorithms that have built-in feature selection methods. For instance, Lasso and RF have their own feature selection methods.