In ML why selecting the best variables?

Question

Almost all ML notebooks out there have a section where they select the best features to use in the model. Why is this step always there ? How bad can it be to keep a variable that is not correlated with the response variable ? If you are really unlucky then yes a feature that is positively correlated with your response in your training set could in fact be negatively correlated with it in the real world. But then, it's not even sure that one will be able to catch it with a feature selection routine.

My assumption is that it used to be a necessary step when computing resources were scarce, but with today's resources it is basically irrelevant.

What is your view ? Can you give a real world example where it would harm the model to keep all training features ?

Answer 1

You are right. If someone is using regularization correctly and doing hyperparameter tuning to avoid overfitting, then it should not be a problem theoretically (ie multi-collinearity will not reduce model performance).

However, it may matter in a number of practical circumstances. Here are two examples:

1. You want to limit the amount of data you need to store in a database for a model that you are frequently running, and it can be expensive storage-wise, and computation-wise to keep variables that don't contribute to model performance. Therefore, I would argue that although computing resources are not 'scarce' they are still monetarily expensive, and using extra resources if there is a way to limit them is also a time sink.
2. For interpretation's sake, it is easier to understand the model if you limit the number of variables. Especially if you need to show stakeholders (if you work as a data scientist) and need to explain model performance.

Answer 2

The feature selection step is there to guard against model overfitting. The feature selection step may decide that all the variables in the dataset are relevant, or it may decide to remove some. If no feature selection step is performed then no variables are removed and the resulting model may be well-fitted but may be (and likely is) overfitted.

The main concern with overfitted models is their poor performance on out-of-sample or validation data. That is the main reason given by Wikipedia and matches my experience. If you have extra variables in the model then irrelevant data will corrupt your models output and make its predictions less accurate.

You could take a more philosophical perspective and say that you want to base your modelling on some consistent and reasonable foundation of statistical reasoning. Whichever system you choose (classical, Bayesian, etc), it is likely to encode some form of Occam's razor and therefore have some mechanism of feature selection built in. That is it say, when comparing two models with the same quality of fit it will prefer the one with the fewest variables selected.

In some cases, the ability of humans to interpret the model is important, as fractalnature points out. Sometimes you want a model so simple that a human can apply it themselves with a calculator. The introduction to the paper Supersparse linear integer models for optimized medical scoring systems provides a nice example of this.

Answer 3

Generally I think we want to get rid of unnecessary/bad features to save ourselves from the curse of dimensionality -- the more features we use, the more data we need to make sure each part of the feature space has enough data to fit the model. On top of that there are concerns specific to our model choice, for example,

1. Random Forest / GBDT. If we have 30 features and set feature_bagging to 10, it takes >= 30C10 = 30,045,015 trees to go through all possibilities. Also, features that are highly linearly correlated with one another do not add extra value to the model but are more possible to be chosen during feature bagging.

2. Distance-based model like K-means can be volunerable to high dimensions.

Answer 4

My assumption is that it used to be a necessary step when computing resources were scarce, but with today's resources it is basically irrelevant.

You are entirely right. In the early days of computing, when resources were scarce, it was necessary just to keep important features and discard the rest.

However, with the current abundance of resources, that is no longer necessary. The model will not overfit more with "useless" features included. Indeed, there is a branch of Machine Learning called "reservoir computing" where you purposely generate random features in the hope that a subset of them are still useful for the model.

For instance, you can checkout the algorithm called "Rocket" for time series prediction.