Let's say I have m training examples with n continuous explanatory variables x1, x2,..., xn and a label y. I'm doing a linear regression.

Is there a way to rank what combinations of explanatory variables are actually the most useful to "predict" y ?

For instance, is there an algorithm to tell me what are the best 2 explanatory variables if I only want to use 2 of them? Or to tell me out of the 2^n possible selections of explanatory variables, which ones are the best ? and which explanatory variables are useless / redudant ?



1 Answer 1


Feature selection is one of the hardest and at the same time important step in creating successful models in general. For you case, you've some number of options aside from exhaustive search (which is doable depending on the number of features and training samples) that are greedy algorithms trying to come up some decent combination of features, namely; forward/backward selection and stepwise selection. Also try Ridge/Lasso regression. Take a look at chapter chapter 6 of Hastie-Tibshirani's: An Introduction to Statistical Learning


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.