1
$\begingroup$

Below formulas, L1 and L2 regularization

enter image description here

enter image description here

Many experts said that L1 regularization makes low-value features zero because of constant value. However, I think that L2 regularization could also make zero value. Could you please explain the reason why L1 has more tendency to make zero value? (It's gonna be great if you let me know this reason by using formula(like above equation)!

$\endgroup$
2
$\begingroup$

The penalty on coefficients of L1 is more aggressive on values close to zero than it is for L2.

With L1, when a weight value comes closer to zero, it tends to get even closer, because of the $\epsilon\lambda$ penalty which stays constant.

With L2, the $\epsilon\lambda w^{(t)}$ term gets smaller, so the regularization gets smaller as a weight comes closer to zero, with the update depending only on $\epsilon\Delta E(w)$.

So the main argument for this to my knowledge is that $$ \lim_{w\rightarrow0}\lambda = \lambda\\ \lim_{w\rightarrow0}\lambda w = 0 $$

$\endgroup$
1
  • $\begingroup$ Thanks!! 'more aggressive' -> Clear explanation!! $\endgroup$ – douner Jan 4 '20 at 7:40
0
$\begingroup$

In my case what makes sense is that for the derivative of an absolute value (L1) forces values to be cero, while for L2(quadratic), the slope of the derivative is smoother.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.