l1 regularization increases sparsity, so unimportant weights are decreased closer to 0. In Deep Learning models, the input usually consists of thousands or millions of features/pixels, and the network usually contains millions to even billions of weights.

Intuitively and theoretically, such feature selection should be very helpful in Deep Learning models to reduce overfitting problems since not all features/weights are important, selecting important ones from millions of weights reduces the function complexity, which therefore reduces the possibility of "memorizing" the training set, which also forces the network to learn more from the important features rather than some useless relations.

However, I have been reading papers on Deep Learning, mostly Computer Vision, it seems like l1 is rarely used in those famous/strong/SOTA architectures and algorithms other than pruning related papers. In contrast, l2 is used in most of them (through weight decay, which is equivalent to l2 when using SGD).

Is there any reason behind this?


Derivative of $L1$ regularization is much more computationally expensive than derivate of $L2$ regularization which is required during backprop.

Also $L1$ regularization causes to sparse feature vector which is not desired in most of the cases.

  • 1
    $\begingroup$ Hi, thanks for answering, could you please explain a little more in detail why a sparse feature vector is not desired in most cases? If we only perform l1 regularization in hidden layers, it will only create sparsity in hidden neurons, which will not directly influence output, why is such sparsity not desired in most of the cases? $\endgroup$
    – seermer
    Aug 3 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.