2
$\begingroup$

l1 regularization increases sparsity, so unimportant weights are decreased closer to 0. In Deep Learning models, the input usually consists of thousands or millions of features/pixels, and the network usually contains millions to even billions of weights.

Intuitively and theoretically, such feature selection should be very helpful in Deep Learning models to reduce overfitting problems since not all features/weights are important, selecting important ones from millions of weights reduces the function complexity, which therefore reduces the possibility of "memorizing" the training set, which also forces the network to learn more from the important features rather than some useless relations.

However, I have been reading papers on Deep Learning, mostly Computer Vision, it seems like l1 is rarely used in those famous/strong/SOTA architectures and algorithms other than pruning related papers. In contrast, l2 is used in most of them (through weight decay, which is equivalent to l2 when using SGD).

Is there any reason behind this?

Improve this question
$\endgroup$
2
  • $\begingroup$ In what way do you think $L1$ regularization performs feature selection? $\endgroup$
    – Dave
    Mar 18, 2022 at 1:50
  • $\begingroup$ @Dave from what I understand, L1 regularization causes sparsity, which means those "unimportant" features will be associated with less weight and toward 0, which makes "important" features stand out. In fact, L1 is commonly used for feature selection in more traditional machine learning. Popular frameworks like scikit-learn provide functions that use l1 for feature selection, You can search on google for "l1 regularization feature selection" to find more related info and more detailed reasonings. $\endgroup$
    – seermer
    Mar 18, 2022 at 16:48

1 Answer 1

Reset to default
1
$\begingroup$

Derivative of $L1$ regularization is much more computationally expensive than derivate of $L2$ regularization which is required during backprop.

Also $L1$ regularization causes to sparse feature vector which is not desired in most of the cases.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Hi, thanks for answering, could you please explain a little more in detail why a sparse feature vector is not desired in most cases? If we only perform l1 regularization in hidden layers, it will only create sparsity in hidden neurons, which will not directly influence output, why is such sparsity not desired in most of the cases? $\endgroup$
    – seermer
    Aug 3, 2021 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.