I'm a bit confused about the superiority of Feature Selection over Feature Engineering or vice versa.
Let's say I just want to get the best possible performance on a couple of models like a neural network, something tree-based and a Naive Bayes Classifier.
Before starting any training I looked at my features and engineered some additional (hopefully) even more expressive features. I did this from a domain expert point of view. For instance I added a new ratio feature
C = A / B because I think this will be a very expressive information for the model.
Furthermore I added several features measuring basically the same thing but in different ways. Lets say a feature
D measuring the the length of any text including empty lines and an additional feature
E measures the length of any text excluding empty lines.
So this leads to very many features in my data set with also very high correlation / multi-collinearity. (of course
E are very high correlated and
C are also high correlated / multi-collinear.
So any kind of correlated based feature selection (between the features) would remove a lot of the engineered features, but could the removal provide the model any kind of better discriminative power with just less information? What helps the model more, keeping all features or removing correlated ones?