Conducting a linear regression model using a loss function, why should I use $L_1$ instead of $L_2$ regularization?

Is it better at preventing overfitting? Is it deterministic (so always a unique solution)? Is it better at feature selection (because producing sparse models)? Does it disperse the weights among the features?


Basically, we add a regularization term in order to prevent the coefficients to fit so perfectly to overfit.

The difference between L1 and L2 is L1 is the sum of weights and L2 is just the sum of the square of weights.

L1 cannot be used in gradient-based approaches since it is not-differentiable unlike L2

L1 helps perform feature selection in sparse feature spaces.Feature selection is to know which features are helpful and which are redundant.

The difference between their properties can be summarized as :

l1 vs l2

  • 1
    $\begingroup$ It isn't true that "L1 cannot be used in gradient-based approaches". Keras supports it, for instance. Yes, the derivative is always constant so it makes it harder for gradient descent to find the minimum. But regularization is a small term within the loss function, so it's not very important in the grand scheme of things. $\endgroup$ Oct 19 '17 at 18:49

L2 has one very important advantage to L1, and that is invariance to rotation and scale.

This is especially important in geographical / physical application.

Say your technician accidentally installed your sensor in 45 degree angel, L1 would be affected, while L2 (Euclidean distance) would remain the same.

  • 4
    $\begingroup$ This is not at all an answer to the question. $\endgroup$
    – kbrose
    Oct 14 '17 at 3:16
  • $\begingroup$ Could you explain the invariance, please? $\endgroup$ Oct 18 '17 at 5:41
  • $\begingroup$ @Chati, the question is about regularization. You are confusing it with other uses of 1-norm and 2-norm in loss functions. $\endgroup$ Oct 19 '17 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.