How do Bayesian methods do automatic feature selection?

Question

Someone asked me this question and I do not know I answered it correctly. I answered the question in the following way: One type of Bayesian method is Bayesian inference and feature selection has to do with ${L}^{1}$ regularization because it is used extensively for this purpose. So, for ${L}^{1}$ regularization, the penalty $\alpha \Omega (\boldsymbol{w}) = \alpha \sum_{i} |w_{i}|$ used to regularize a cost function is equivalent to the log-prior term that is maximized by MAP Bayesian inference when the prior is an isotropic Laplace distribution.

But my question is this an automatic feature selection? ${L}^{1}$ regularization finds the specific subset of the available features to be used. Also is my answer correct to this question? I am just curious to know if my line of thinking makes sense or if it does not. If it does not, please let me know why. Thanks

Answer 1

Your interpretation of L1 regularization sounds right. Is it used to perform feature selection? Yes, in the broad sense that this 'encourages' coefficients in the linear model to be 0, and those features with 0 coefficients are not used and can be removed.

Of course, this assumption about the prior distribution of coefficients is just an assumption you're adding. You are asserting that you need a lot of evidence to believe the coefficients are nonzero. I'm not sure it's the L1 regularization that's doing the feature selection, but your assumption, implemented by L1 regularization, that's leading you to conclude that some features do not contribute to the model.

Answer 2

From a bayesian perspective, I would not begin the answer talking about the regularization term. I would rather say what is the prior distribution you are assuming (since this is the starting point in the bayesian framework) and then the regularization term comes up as a consequence, after calculating the log-posterior.

Also, I believe that feature selection is called "automatic" in the sense that the regularization term may drive to zero some weights when you minimize the cost function, without the need of selecting the features in a previous step (in practice you may need to do it anyway.)