Can I expect necessarily higher accuracy when feature selection/dimension reduction is used to select a subset of features?

Question

The feature selection/dimension reduction is performed to eliminate irrelevant or redundant features so it will improve the computation efficiency (less computationally expensive). My question is that can we expect any changes in the accuracy of prediction when efficient features are used for classification vs when all features used?

Answer 1

Using all features would usually lead to over-fitting, that is your model would not generalise well to unseen data. To overcome this, while using as much information possible we resort to feature selection ( sometimes feature generation ), and dimensionality reduction. Dimensionality reduction techniques like Principal Component Analysis (PCA) tries to find new features that explain maximum variability in data, using the existing features. In this way, we ideally reduce the number of features in the model, while making sure most of the variability is explained.

So as a result:

1. This new model is computationally faster since we are no longer using all the features and burdening our model.
2. Generalises well since we are not over-fitting leading to greater accuracy on unseen data.

Answer 2

It's a trade-off

Information is in the variance of data

Let's say, we have a dataset with very high dimensionality, definitely, it will create a problem for any model(may refer the logic behind Curse of Dimensionality)

- We lose some variance while reducing the dimension
- We helped the model learning the data. There might be other underlying reasons too (than computation) i.e. removal of colinear and irrelevant features

So, if the gain is more than the loss, then it will definitely improve the model's performance.

Though it was not your question but be mindful of the fact that with Feature engineering we also create some good Feature by doing an intelligent Exploratory data analysis

I made this point because with dataset not having too many Features(when not considered as very high dimensionality), Feature engineering is more about finding new useful feature using the info from the data and some real-world knowledge about the problem.
Manytime, a simple transformation using some Heuristics also works(Kaggle kernels)