32
$\begingroup$

In Keras, there are 2 methods to reduce over-fitting. L1,L2 regularization or dropout layer.

What are some situations to use L1,L2 regularization instead of dropout layer? What are some situations when dropout layer is better?

$\endgroup$
2
  • $\begingroup$ L1, L2 regularizers don't really work in NN's if you are using an NN..A good intuition is given by lec 2/3/4 (I am not sure) of Stanford CNN course.. Just serach it on utube $\endgroup$
    – DuttaA
    Commented Aug 23, 2018 at 17:36
  • $\begingroup$ I would say that there aren't in Keras, but rather on ML, Statistics etc. $\endgroup$
    – Enzo Dtz
    Commented Jul 5, 2022 at 2:02

2 Answers 2

25
$\begingroup$

I am unsure there will be a formal way to show which is best in which situations - simply trying out different combinations is likely best!

It is worth noting that Dropout actually does a little bit more than just provide a form of regularisation, in that it is really adding robustness to the network, allowing it to try out many many different networks. This is true because the randomly deactivated neurons are essentially removed for that forward/backward pass, thereby giving the same effect as if you had used a totally different network! Have a look at this post for a few more pointers regarding the beauty of dropout layers.

$L_1$ versus $L_2$ is easier to explain, simply by noting that $L_2$ treats outliers a little more thoroughly - returning a larger error for those points. Have a look here for more detailed comparisons.

$\endgroup$
5
  • $\begingroup$ When I use L2 regularization, the loss rate increase. Do you know why? $\endgroup$
    – N.IT
    Commented Sep 5, 2018 at 11:44
  • $\begingroup$ If your $L2$ loss increases, so would your $L1$ loss (just perhaps more slowly). Either way, your model is diverging from a minimum in the loss curve i.e. it isn't learning. You might want to think about other parts of your model again, such as the architecture, data preprocessing or class-imbalance in your data. Those keywords might help you in the right direction with some searches. $\endgroup$
    – n1k31t4
    Commented Sep 5, 2018 at 13:15
  • $\begingroup$ Also, dropout can cause validation accuracy sometimes to be higher than train accuracy, which is indicative of a good performance. $\endgroup$ Commented Jun 6, 2020 at 2:47
  • $\begingroup$ @AnshumanKumar which is indicative of a good performance. Can you elaborate this a bit more? $\endgroup$ Commented Apr 13, 2021 at 19:49
  • $\begingroup$ can we use L1 regularization and dropout together? $\endgroup$
    – MUK
    Commented May 16, 2022 at 12:06
5
$\begingroup$

It seems deciding between L2 and Dropout is a "guess and check" type of thing, unfortunately. Both are used to make the network more "robust" and reduce overfitting by preventing the network from relying too heavily on any given neuron. ie: it is generally believed that it would be better to have many neurons contributing to a model’s output, rather than a select few. L2 and Dropout do this by different means, so you really have to play around to see what gives you a different result.

Dropout randomly mutes some percentage of neurons (provided by you) each forward pass through the network, forcing the network to diversify.

L2 reduces the contribution of high outlier neurons (those significantly larger than the median) and prevents any one neuron from exploding. This also forces the network to diversify.

L1 should really be in its own category, as it is most useful for features selection and small networks. It almost does the opposite of L2 and Dropout by simplifying the network and muting nome neurons.

If you notice that adding a small regularization decreases your accuracy / increases your loss, it's probably because your network was overfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.