Is there any feature selection method that works especially well for regressions?

I used backwards elimination and forward selection before a lot but I've recently read that even though it's historical, it's not correct... but maybe a variance threshold? Correlation threshold? Chi square? I've been searching but everywhere I look I find a new way of doing it... so I don't get a definitive answer

Some context: I'm running a multinomial logit, with 3 classes on the y, and the regressors are a mix of demographics (age, sex, years at job, income, mortgage, debt to income ratio, etc), loan specific variables (loan amount, loan term, APR, rep APR) and some "time" variables (amount of days that client spent in certain status, how many times client entered the status). My objective is to predict.

  • $\begingroup$ Specific for regression as opposed to what? $\endgroup$
    – Dave
    Oct 10, 2020 at 11:56
  • $\begingroup$ Did you try lasso/ridge? $\endgroup$
    – Peter
    Oct 10, 2020 at 13:37
  • $\begingroup$ It sounds like you have a categorical dependent variable, so why do you ask about a regression-specific method? $\endgroup$
    – Adrià Luz
    Sep 24, 2021 at 21:16

1 Answer 1


Lasso or L1 regression is a feature selection method specific to regression.

It adds an additional penalty term to the loss function for extreme coefficient values. The result is that some of the coefficients shrink to zero and the value of those features do not contribute to the prediction.

  • $\begingroup$ I disagree that they are removed from the model. They absolutely contribute to the model through the penalized loss, even if their coefficients wind up as zero. If you run OLS on the variables whose LASSO coefficients are not zero (the survivors, so to speak), you will (or at least can) get different coefficient estimates from OLS than from LASSO. Ditto for logistic regression. $\endgroup$
    – Dave
    Sep 24, 2021 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.